skip to main content
research-article

Pithos: Distributed Storage for Massive Multi-User Virtual Environments

Published:12 July 2017Publication History
Skip Abstract Section

Abstract

There has been significant research effort into peer-to-peer (P2P) massively multi-user virtual environments (MMVEs). A number of architectures have been proposed to implement the P2P approach; however, the development of fully distributed MMVEs has met with a number of challenges. In this work, we address one of the key remaining challenges of state consistency and persistency in P2P MMVEs. Having reviewed state management and persistency architectures currently receiving research attention, we have identified deficiencies such as lack of load balancing, responsiveness, and scalability. To address these deficiencies, we present Pithos—a reliable, responsive, secure, load-balanced, and scalable distributed storage system, suited to P2P MMVEs. Pithos is designed specifically for P2P MMVEs, and we show that it improves the reliability and responsiveness of storage architectures as compared to existing P2P state persistency architectures.

Pithos is implemented as an OverSim simulation running on the OMNeT++ network simulation framework. It is evaluated using up to 10,400 peers, with realistic latency profiles, with up to 15.8 million storage and retrieval requests that are generated to store a total of 2.4 million objects. Each peer in Pithos uses a maximum of 1,950Bps bandwidth to achieve 99.98% storage reliability, while the most reliable overlay storage configuration tested only achieved 93.65% reliability, using 2,182Bps bandwidth. Pithos is also more responsive than overlay storage, with an average responsiveness of 0.192s, compared with the average overlay responsiveness of 1.4s when retrieving objects from storage.

Skip Supplemental Material Section

Supplemental Material

References

  1. Franz Aurenhammer. 1991. Voronoi diagrams—A survey of a fundamental geometric data structure. Comput. Surv. 23, 3 (1991), 345--405. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Richard Allan Bartle. 2004. Designing Virtual Worlds. New Riders.Google ScholarGoogle Scholar
  3. Nathaniel E. Baughman, Marc Liberatore, and Brian Neil Levine. 2007. Cheat-proof playout for centralized and peer-to-peer gaming. IEEE/ACM Trans.Netw. 15, 1 (2007), 1--13. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Ingmar Baumgart, Bernhard Heep, and Stephan Krause. 2007. OverSim: A flexible overlay network simulation framework. In IEEE Global Internet Symposium (GI) at INFOCOMM. 79--84. Google ScholarGoogle ScholarCross RefCross Ref
  5. Abhijit Belapurkar, Anirban Chakrabarti, Harigopal Ponnapalli, Niranjan Varadarajan, Srinivas Padmanabhuni, and Srikanth Sundarrajan. 2009. Distributed Systems Security: Issues, Processes and Solutions. Wiley. Google ScholarGoogle ScholarCross RefCross Ref
  6. Ashwin Bharambe, John R. Douceur, Jacob R. Lorch, Thomas Moscibroda, Jeffrey Pang, Srinivasan Seshan, and Xinyu Zhuang. 2008. Donnybrook: Enabling large-scale, high-speed, peer-to-peer games. SIGCOMM Comput. Commun. Rev. 38, 4 (Aug. 2008), 389--400. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Ashwin Bharambe, Jeffrey Pang, and Srinivasan Seshan. 2006. Colyseus: A distributed architecture for online multiplayer games. In Proceedings of the 3rd Conf. on Networked Systems Design 8 Implementation (NSDI’06), Vol. 3.Google ScholarGoogle Scholar
  8. Eliya Buyukkaya and Maha Abdallah. 2008. Data management in voronoi-based p2p gaming. In Proceedings of the Consumer Communications and Networking Conference (CCNC’08). 1050--1053. Google ScholarGoogle ScholarCross RefCross Ref
  9. Eliya Buyukkaya, Maha Abdallah, and Gwendal Simon. 2015. A survey of peer-to-peer overlay approaches for networked virtual environments. Peer-to-Peer Netw. Appl. 8, 2 (2015), 276--300. Google ScholarGoogle ScholarCross RefCross Ref
  10. James Calvin, Alan Dickens, Bob Gaines, Paul Metzger, Dale Miller, and Owen Owen. 1993. The SIMNET virtual world architecture. In Proceedings of IEEE Virtual Reality Annual International Symposium. 450--455. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Miguel Castro, Peter Druschel, Ayalvadi Ganesh, Antony Rowstron, and Dan S. Wallach. 2002. Secure routing for structured peer-to-peer overlay networks. ACM SIGOPS Oper. Syst. Rev. 36, SI (Dec. 2002), 299--314.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Shao-Chen Chang. 2008. Voronoi Diagram Based State Management for Peer-to-Peer Virtual Environments. Master’s thesis. National Central University.Google ScholarGoogle Scholar
  13. Fang Chen and Vana Kalogeraki. 2005. Adaptive real-time update dissemination in distributed virtual simulation environments. In Proceedings of the 8th IEEE Intl.Symposium on Object-Oriented Real-Time Distributed Computing (ISORC’05). 233--236. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Ian Clarke, Oskar Sandberg, Brandon Wiley, and Theodore W. Hong. 2001. Freenet: A distributed anonymous information storage and retrieval system. In Proceedings of the International Workshop on Designing Privacy Enhancing Technologies. 46--66. Google ScholarGoogle ScholarCross RefCross Ref
  15. Mark Claypool. 2005. The effect of latency on user performance in real-time strategy games. Comput. Netw. 49, 1 (Sept. 2005), 52--70. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Scott Douglas, Egemen Tanin, Aaron Harwood, and Shanika Karunasekera. 2005. Enabling massively multi-player online gaming applications on a P2P architecture. In Proceedings of the IEEE International Conference on Information and Automation (ICIA’07). 7--12.Google ScholarGoogle Scholar
  17. Peter Druschel and Antony Rawstron. 2001. PAST: A large-scale, persistent peer-to-peer storage utility. In Proceedings of the 8th Workshop on Hot Topics in Operating Systems (HotOS’01). Google ScholarGoogle ScholarCross RefCross Ref
  18. Herman Arnold Engelbrecht and Gregor Schiele. 2013. Koekepan: Minecraft as a research platform. In Proceedings of the 2013 12th Annual Workshop on Network and Systems Support for Games (NetGames’13). 1--3. Google ScholarGoogle ScholarCross RefCross Ref
  19. Lu Fan. 2009. Solving Key Design Issues for Massively Multiplayer Online Games on Peer-to-Peer Architectures. Ph.D. Dissertation. School of Mathematical and Computer Sciences—Heriot-Watt University.Google ScholarGoogle Scholar
  20. Lu Fan, Hamish Taylor, and Phil Trinder. 2007. Mediator: A design framework for P2P MMOGs. In Proceedings of the 6th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’07). 43--48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Lu Fan, Phil Trinder, and Hamish Taylor. 2010. Design issues for peer-to-peer massively multiplayer online games. Intl. J. Adv. Media Commun. 4, 2 (2010), 108--125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Stefano Ferretti. 2008. A synchronization protocol for supporting peer-to-peer multiplayer online games in overlay networks. In Proceedings of the 2nd International Conference on Distributed Event-based Systems (DEBS’08). ACM, New York, NY, 83--94. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Luca Ferrucci, Laura Ricci, Michele Albano, Ranieri Baraglia, and Matteo Mordacchini. 2016. Multidimensional range queries on hierarchical Voronoi overlays. J. Comput. System Sci. 82, 7 (2016), 1161--1179. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Cooperative Association for Internet Data Analysis. 2012. The CAIDA UCSD Macroscopic Topology Dataset. Retrieved from http://www.caida.org/tools/measurement/skitter/.Google ScholarGoogle Scholar
  25. Brendan J. Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. Science 315, 5814 (2007), 972--976. Google ScholarGoogle ScholarCross RefCross Ref
  26. Davide Frey, Jérôme Royan, Romain Piegay, Anna-Marie Kermarrec, Emmanuelle Anceaume, and Fabrice Le Fessant. 2008. Solipsis: A decentralized architecture for virtual environments. In Proceedings of the 1st International Workshop on Massively Multiuser Virtual Environments (MMVE’08). 29--33.Google ScholarGoogle Scholar
  27. Chris Gauthier Dickey, Daniel Zappala, and Virginia Lo. 2004. A fully distributed architecture for massively multiplayer online games. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’04). 171--171.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Laurent Gautier and Christophe Diot. 1998. Design and evaluation of MiMaze, A multi-player game on the internet. In Proceedings of the IEEE Intl. Conf. on Multimedia Computing and Systems (ICMCS’98). IEEE Computer Society, Washington, DC, USA, 233. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. John Sebastian Gilmore and Herman Arnold Engelbrecht. 2011. Pithos: A state persistency architecture for peer-to-peer massively multiuser virtual environments. In Proceedings of the IEEE Intl. Workshop on Haptic Audio Visual Environments and Games (HAVE’11). 1--6. Google ScholarGoogle ScholarCross RefCross Ref
  30. John Sebastian Gilmore and Herman Arnold Engelbrecht. 2012. A survey of state persistency in peer-to-peer massively multiplayer online games. IEEE Trans. Parallel Distrib. Syst. 23, 5 (May 2012), 818--834. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. John Sebastian Gilmore and Herman Arnold Engelbrecht. 2015. Predicting object lifetimes in finite distributed storage systems under churn. In Proceedings of the 7th ACM Intl. Workshop on Massively Multiuser Virtual Environments (MMVE’15). ACM, New York, NY, 1--6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Thorsten Hampel, Thomas Bopp, and Robert Hinn. 2006. A peer-to-peer architecture for massive multiplayer online games. In Proceedings of 5th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’06). 48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Aaron Harwood and Egemen Tanin. 2003. Hashing spatial content over peer-to-peer networks. In Proceedings of the Australian Telecommunications, Networks and Applications Conference (ATNAC’03). 1--5.Google ScholarGoogle Scholar
  34. Ragib Hasan, Zahid Anwar, William Yurcik, Larry Brumbaugh, and Roy Campbell. 2005. A survey of peer-to-peer storage techniques for distributed file systems. In Proceedings of the Intl. Conf. on Information Technology: Coding and Computing (ITCC), Vol. 2. 205--213. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Pedro Henriques and Adolfo Garo. 2012. A Lightweight Distributed Super Peer Election: Algorithm for Unstructured Dynamic P2P Systems on the Internet of Things. LAP Lambert Academic Publishing, Germany.Google ScholarGoogle Scholar
  36. Sebastian Holzapfel, Sebastian Schuster, and Torben Weis. 2011. VoroStore -- A secure and reliable data storage for peer-to-peer-based MMVEs. In Proceedings of the 2011 IEEE 11th International Conference on Computer and Information Technology. 35--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Shun-Yun Hu, Shao-Chen Chang, and Jehn-Ruey Jiang. 2008. Voronoi state management for peer-to-peer massively multiplayer online games. In Proceedings of the Consumer Communications and Networking Conference (CCNC’08). 1134--1138. Google ScholarGoogle ScholarCross RefCross Ref
  38. Shun-Yun Hu, Jui-Fa Chen, and Tsu-Han Chen. 2006. VON: A scalable peer-to-peer network for virtual environments. IEEE Netw. 20, 4 (2006), 22--31. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Shun-Yun Hu and Kuan-Ta Chen. 2011. Self-organizing spatial publish subscribe. In Proceedings of the 8th ACM International Conference on Autonomic computing (ICAC). ACM, 171--172. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Takuji Iimura, Hiroaki Hazeyama, and Youki Kadobayashi. 2004. Zoned federation of game servers: A peer-to-peer approach to scalable multi-player online games. In Proceedings of the 3rd ACM SIGCOMM Workshop on Network and System Support for Games (NetGames). 116--120. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Jared Jardine and Daniel Zappala. 2008. A hybrid architecture for massively multiplayer online games. In Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames). 60--65. Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. Hanna Kavalionak, Emanuele Carlini, Laura Ricci, Alberto Montresor, and Massimo Coppola. 2015. Integrating peer-to-peer and cloud computing for massively multiuser online games. Peer-to-Peer Networking and Applications 8, 2 (2015), 301--319. Google ScholarGoogle ScholarCross RefCross Ref
  43. Björn Knutsson, Honghui Lu, Wei Xu, and Bryan Hopkins. 2004. Peer-to-peer support for massively multiplayer games. In Proceedings of the 23rd Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM), Vol. 1. 107. Google ScholarGoogle ScholarCross RefCross Ref
  44. Santosh Kulkarni. 2009. Badumna network suite: A decentralized network engine for massively multiplayer online applications. In Proceedings of the IEEE 9th International Conference on Peer-to-Peer Computing (P2P). 178--183. Google ScholarGoogle ScholarCross RefCross Ref
  45. Meirong Liu, Erkki Harjula, and Mika Ylianttila. 2013. An efficient selection algorithm for building a super-peer overlay. Journal of Internet Services and Applications 4, 1 (2013), 4.Google ScholarGoogle ScholarCross RefCross Ref
  46. Eng Keong Lua, J. Crowcroft, M. Pias, R. Sharma, and S. Lim. 2005. A survey and comparison of peer-to-peer overlay network schemes. IEEE Communications Surveys Tutorials 7 (2nd quarter 2005), 72--93.Google ScholarGoogle Scholar
  47. Martin Mauve, Jürgen Vogel, Volker Hilt, and Wolfgang Effelsberg. 2004. Local-lag and timewarp: Providing consistency for replicated continuous applications. IEEE Trans. on Multimedia 6, 1 (Feb 2004), 47--57. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Petar Maymounkov and David Mazières. 2002. Kademlia: A peer-to-peer information system based on the XOR metric. In Revised Papers from the 1st International Workshop on Peer-to-Peer Systems (IPTPS) (IPTPS’01). Springer-Verlag, London, UK, UK, 53--65. Google ScholarGoogle ScholarCross RefCross Ref
  49. Madjid Merabti and Abdennour El Rhalibi. 2004. Peer-to-peer architecture and protocol for a massively multiplayer online game. In GlobeCom Workshops. 519--528. Google ScholarGoogle ScholarCross RefCross Ref
  50. Alberto Montresor. 2004. A robust protocol for building superpeer overlay topologies. In Proceedings of the 4th International Conference on Peer-to-Peer Computing. IEEE, Zurich, Switzerland, 202--209. Google ScholarGoogle ScholarCross RefCross Ref
  51. Alberto Montresor and Jelasity Márk. 2009. PeerSim: A scalable P2P simulator. In Proceedings of the 2009 IEEE 9th International Conference on Peer-to-Peer Computing. 99--100. Google ScholarGoogle ScholarCross RefCross Ref
  52. Matteo Mordacchini, Laura Ricci, Luca Ferrucci, Michele Albano, and Ranieri Baraglia. 2010. Hivory: Range queries on hierarchical voronoi overlays. In Proceedings of the 2010 IEEE 10th International Conference on Peer-to-Peer Computing (P2P). 1--10. Google ScholarGoogle ScholarCross RefCross Ref
  53. Marvin Rausand and Arnljot Høyland. 2004. System Reliability Theory: Models, Statistical Methods, and Applications. Wiley-Interscience.Google ScholarGoogle Scholar
  54. Antony Rawstron and Peter Druschel. 2001. Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems. In Proceedings of the 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware). 329--350. Google ScholarGoogle ScholarCross RefCross Ref
  55. Laura Ricci and Luca Genovali. 2010. State management in distributed virtual environments: A voronoi base approach. In Proceedings of the International Congress on Ultra Modern Telecommunications and Control Systems. 881--887. Google ScholarGoogle ScholarCross RefCross Ref
  56. Sean Rooney, Daniel Bauer, and Rudy Deydier. 2004. A federated peer-to-peer network game architecture. IEEE Commun. Mag. 42, 5 (2004), 114--122. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Antony Rowstron and Peter Druschel. 2001. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. ACM SIGOPS Operat. Syst. Rev. 35, 5 (2001), 188--201. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Sebastian Schuster and Torben Weis. 2011. Enforcing game rules in untrusted P2P-based MMVEs. In Proceedings of the 4th International ICST Conference on Simulation Tools and Techniques (SIMUTools’11). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, 288--295.Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. Nathan Sheldon, Eric Girard, Seth Borg, Mark Claypool, and Emmanuel Agu. 2003. The effect of latency on user performance in Warcraft III. In Proceedings of the 2nd Workshop on Network and System Support for Games (NetGames’03). 3--14. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Moritz Steiner, Taoufik En-Najjary, and Ernst W. Biersack. 2007. A global view of KAD. In Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement (IMC’07). ACM, New York, NY, 117--122. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan. 2001. Chord: A scalable peer-to-peer lookup service for internet applications. SIGCOMM Comput. Commun. Rev. 31, 4 (2001), 149--160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Pin-Yun Tarng, Kuan-Ta Chen, and Polly Huang. 2008. An analysis of WoW players’ game hours. In Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’08). ACM, 47--52. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. Amin Vahdat, Ken Yocum, Kevin Walsh, Priya Mahadevan, Dejan Kostić, Jeff Chase, and David Becker. 2002. Scalability and accuracy in a large-scale network emulator. ACM SIGOPS Operat. Syst. Rev. 36, SI (Dec. 2002), 271--284.Google ScholarGoogle ScholarDigital LibraryDigital Library
  64. András Varga. 2012. OMNeT++. Retrieved from http://www.omnetpp.org/.Google ScholarGoogle Scholar
  65. Matteo Varvello. 2009. A Peer-to-Peer Architecture for Networked Virtual Environments. Ph.D. Dissertation. Telecom Paristech.Google ScholarGoogle Scholar
  66. Matteo Varvello, Christophe Diot, and Ernst W. Biersack. 2009a. P2P second life: Experimental validation using kad. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM’09). 1161--1169.Google ScholarGoogle Scholar
  67. Matteo Varvello, Christophe Diot, and Ernst W. Biersack. 2009b. A walkable kademlia network for virtual worlds. In Proceedings of the IEEE International Conference on Computer Communications Workshops (INFOCOM’09). 1--2. Google ScholarGoogle ScholarCross RefCross Ref
  68. Matteo Varvello, Fabio Picconi, Christophe Diot, and Ernst Biersack. 2008. Is there life in second life?. In Proceedings of the 2008 ACM CoNEXT Conference (CoNEXT’08). ACM, New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  69. VAST Development Team. 2010. VAST. Retrieved from vast.sourceforge.netGoogle ScholarGoogle Scholar
  70. Jared Winick and Sugih Jamin. 2002. Init-3.0: Internet Topology Generator. Technical Report CSE-TR-456-02. University of Mishigan.Google ScholarGoogle Scholar
  71. Amir Yahyavi and Bettina Kemme. 2013. Peer-to-peer architectures for massively multiplayer online games: A survey. ACM Comput. Surv. 46, 1 (July 2013), Article 9, 51 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  72. Anthony (Peiqun) Yu and Son T. Vuong. 2005. MOPAR: A mobile peer-to-peer overlay architecture for interest management of massively multiplayer online games. In Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV’05). 99--104.Google ScholarGoogle Scholar
  73. Jane Yang Yu and Peter H. J. Chong. 2005. A survey of clustering schemes for mobile ad hoc networks. IEEE Commun. Surv. Tutor. 7 (1st quarter 2005), 32--48.Google ScholarGoogle Scholar
  74. Kaiwen Zhang and Bettina Kemme. 2011. Transaction models for massively multiplayer online games. In Proceedings of the 2011 IEEE 30th International Symposium on Reliable Distributed Systems. 31--40. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Pithos: Distributed Storage for Massive Multi-User Virtual Environments

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!