Abstract
There has been significant research effort into peer-to-peer (P2P) massively multi-user virtual environments (MMVEs). A number of architectures have been proposed to implement the P2P approach; however, the development of fully distributed MMVEs has met with a number of challenges. In this work, we address one of the key remaining challenges of state consistency and persistency in P2P MMVEs. Having reviewed state management and persistency architectures currently receiving research attention, we have identified deficiencies such as lack of load balancing, responsiveness, and scalability. To address these deficiencies, we present Pithos—a reliable, responsive, secure, load-balanced, and scalable distributed storage system, suited to P2P MMVEs. Pithos is designed specifically for P2P MMVEs, and we show that it improves the reliability and responsiveness of storage architectures as compared to existing P2P state persistency architectures.
Pithos is implemented as an OverSim simulation running on the OMNeT++ network simulation framework. It is evaluated using up to 10,400 peers, with realistic latency profiles, with up to 15.8 million storage and retrieval requests that are generated to store a total of 2.4 million objects. Each peer in Pithos uses a maximum of 1,950Bps bandwidth to achieve 99.98% storage reliability, while the most reliable overlay storage configuration tested only achieved 93.65% reliability, using 2,182Bps bandwidth. Pithos is also more responsive than overlay storage, with an average responsiveness of 0.192s, compared with the average overlay responsiveness of 1.4s when retrieving objects from storage.
Supplemental Material
Available for Download
Supplemental movie, appendix, image and software files for, Pithos: Distributed Storage for Massive Multi-User Virtual Environments
- Franz Aurenhammer. 1991. Voronoi diagrams—A survey of a fundamental geometric data structure. Comput. Surv. 23, 3 (1991), 345--405. Google Scholar
Digital Library
- Richard Allan Bartle. 2004. Designing Virtual Worlds. New Riders.Google Scholar
- Nathaniel E. Baughman, Marc Liberatore, and Brian Neil Levine. 2007. Cheat-proof playout for centralized and peer-to-peer gaming. IEEE/ACM Trans.Netw. 15, 1 (2007), 1--13. Google Scholar
Digital Library
- Ingmar Baumgart, Bernhard Heep, and Stephan Krause. 2007. OverSim: A flexible overlay network simulation framework. In IEEE Global Internet Symposium (GI) at INFOCOMM. 79--84. Google Scholar
Cross Ref
- Abhijit Belapurkar, Anirban Chakrabarti, Harigopal Ponnapalli, Niranjan Varadarajan, Srinivas Padmanabhuni, and Srikanth Sundarrajan. 2009. Distributed Systems Security: Issues, Processes and Solutions. Wiley. Google Scholar
Cross Ref
- Ashwin Bharambe, John R. Douceur, Jacob R. Lorch, Thomas Moscibroda, Jeffrey Pang, Srinivasan Seshan, and Xinyu Zhuang. 2008. Donnybrook: Enabling large-scale, high-speed, peer-to-peer games. SIGCOMM Comput. Commun. Rev. 38, 4 (Aug. 2008), 389--400. Google Scholar
Digital Library
- Ashwin Bharambe, Jeffrey Pang, and Srinivasan Seshan. 2006. Colyseus: A distributed architecture for online multiplayer games. In Proceedings of the 3rd Conf. on Networked Systems Design 8 Implementation (NSDI’06), Vol. 3.Google Scholar
- Eliya Buyukkaya and Maha Abdallah. 2008. Data management in voronoi-based p2p gaming. In Proceedings of the Consumer Communications and Networking Conference (CCNC’08). 1050--1053. Google Scholar
Cross Ref
- Eliya Buyukkaya, Maha Abdallah, and Gwendal Simon. 2015. A survey of peer-to-peer overlay approaches for networked virtual environments. Peer-to-Peer Netw. Appl. 8, 2 (2015), 276--300. Google Scholar
Cross Ref
- James Calvin, Alan Dickens, Bob Gaines, Paul Metzger, Dale Miller, and Owen Owen. 1993. The SIMNET virtual world architecture. In Proceedings of IEEE Virtual Reality Annual International Symposium. 450--455. Google Scholar
Digital Library
- Miguel Castro, Peter Druschel, Ayalvadi Ganesh, Antony Rowstron, and Dan S. Wallach. 2002. Secure routing for structured peer-to-peer overlay networks. ACM SIGOPS Oper. Syst. Rev. 36, SI (Dec. 2002), 299--314.Google Scholar
Digital Library
- Shao-Chen Chang. 2008. Voronoi Diagram Based State Management for Peer-to-Peer Virtual Environments. Master’s thesis. National Central University.Google Scholar
- Fang Chen and Vana Kalogeraki. 2005. Adaptive real-time update dissemination in distributed virtual simulation environments. In Proceedings of the 8th IEEE Intl.Symposium on Object-Oriented Real-Time Distributed Computing (ISORC’05). 233--236. Google Scholar
Digital Library
- Ian Clarke, Oskar Sandberg, Brandon Wiley, and Theodore W. Hong. 2001. Freenet: A distributed anonymous information storage and retrieval system. In Proceedings of the International Workshop on Designing Privacy Enhancing Technologies. 46--66. Google Scholar
Cross Ref
- Mark Claypool. 2005. The effect of latency on user performance in real-time strategy games. Comput. Netw. 49, 1 (Sept. 2005), 52--70. Google Scholar
Digital Library
- Scott Douglas, Egemen Tanin, Aaron Harwood, and Shanika Karunasekera. 2005. Enabling massively multi-player online gaming applications on a P2P architecture. In Proceedings of the IEEE International Conference on Information and Automation (ICIA’07). 7--12.Google Scholar
- Peter Druschel and Antony Rawstron. 2001. PAST: A large-scale, persistent peer-to-peer storage utility. In Proceedings of the 8th Workshop on Hot Topics in Operating Systems (HotOS’01). Google Scholar
Cross Ref
- Herman Arnold Engelbrecht and Gregor Schiele. 2013. Koekepan: Minecraft as a research platform. In Proceedings of the 2013 12th Annual Workshop on Network and Systems Support for Games (NetGames’13). 1--3. Google Scholar
Cross Ref
- Lu Fan. 2009. Solving Key Design Issues for Massively Multiplayer Online Games on Peer-to-Peer Architectures. Ph.D. Dissertation. School of Mathematical and Computer Sciences—Heriot-Watt University.Google Scholar
- Lu Fan, Hamish Taylor, and Phil Trinder. 2007. Mediator: A design framework for P2P MMOGs. In Proceedings of the 6th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’07). 43--48. Google Scholar
Digital Library
- Lu Fan, Phil Trinder, and Hamish Taylor. 2010. Design issues for peer-to-peer massively multiplayer online games. Intl. J. Adv. Media Commun. 4, 2 (2010), 108--125. Google Scholar
Digital Library
- Stefano Ferretti. 2008. A synchronization protocol for supporting peer-to-peer multiplayer online games in overlay networks. In Proceedings of the 2nd International Conference on Distributed Event-based Systems (DEBS’08). ACM, New York, NY, 83--94. Google Scholar
Digital Library
- Luca Ferrucci, Laura Ricci, Michele Albano, Ranieri Baraglia, and Matteo Mordacchini. 2016. Multidimensional range queries on hierarchical Voronoi overlays. J. Comput. System Sci. 82, 7 (2016), 1161--1179. Google Scholar
Digital Library
- Cooperative Association for Internet Data Analysis. 2012. The CAIDA UCSD Macroscopic Topology Dataset. Retrieved from http://www.caida.org/tools/measurement/skitter/.Google Scholar
- Brendan J. Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. Science 315, 5814 (2007), 972--976. Google Scholar
Cross Ref
- Davide Frey, Jérôme Royan, Romain Piegay, Anna-Marie Kermarrec, Emmanuelle Anceaume, and Fabrice Le Fessant. 2008. Solipsis: A decentralized architecture for virtual environments. In Proceedings of the 1st International Workshop on Massively Multiuser Virtual Environments (MMVE’08). 29--33.Google Scholar
- Chris Gauthier Dickey, Daniel Zappala, and Virginia Lo. 2004. A fully distributed architecture for massively multiplayer online games. In Proceedings of 3rd ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’04). 171--171.Google Scholar
Digital Library
- Laurent Gautier and Christophe Diot. 1998. Design and evaluation of MiMaze, A multi-player game on the internet. In Proceedings of the IEEE Intl. Conf. on Multimedia Computing and Systems (ICMCS’98). IEEE Computer Society, Washington, DC, USA, 233. Google Scholar
Digital Library
- John Sebastian Gilmore and Herman Arnold Engelbrecht. 2011. Pithos: A state persistency architecture for peer-to-peer massively multiuser virtual environments. In Proceedings of the IEEE Intl. Workshop on Haptic Audio Visual Environments and Games (HAVE’11). 1--6. Google Scholar
Cross Ref
- John Sebastian Gilmore and Herman Arnold Engelbrecht. 2012. A survey of state persistency in peer-to-peer massively multiplayer online games. IEEE Trans. Parallel Distrib. Syst. 23, 5 (May 2012), 818--834. Google Scholar
Digital Library
- John Sebastian Gilmore and Herman Arnold Engelbrecht. 2015. Predicting object lifetimes in finite distributed storage systems under churn. In Proceedings of the 7th ACM Intl. Workshop on Massively Multiuser Virtual Environments (MMVE’15). ACM, New York, NY, 1--6. Google Scholar
Digital Library
- Thorsten Hampel, Thomas Bopp, and Robert Hinn. 2006. A peer-to-peer architecture for massive multiplayer online games. In Proceedings of 5th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’06). 48. Google Scholar
Digital Library
- Aaron Harwood and Egemen Tanin. 2003. Hashing spatial content over peer-to-peer networks. In Proceedings of the Australian Telecommunications, Networks and Applications Conference (ATNAC’03). 1--5.Google Scholar
- Ragib Hasan, Zahid Anwar, William Yurcik, Larry Brumbaugh, and Roy Campbell. 2005. A survey of peer-to-peer storage techniques for distributed file systems. In Proceedings of the Intl. Conf. on Information Technology: Coding and Computing (ITCC), Vol. 2. 205--213. Google Scholar
Digital Library
- Pedro Henriques and Adolfo Garo. 2012. A Lightweight Distributed Super Peer Election: Algorithm for Unstructured Dynamic P2P Systems on the Internet of Things. LAP Lambert Academic Publishing, Germany.Google Scholar
- Sebastian Holzapfel, Sebastian Schuster, and Torben Weis. 2011. VoroStore -- A secure and reliable data storage for peer-to-peer-based MMVEs. In Proceedings of the 2011 IEEE 11th International Conference on Computer and Information Technology. 35--40. Google Scholar
Digital Library
- Shun-Yun Hu, Shao-Chen Chang, and Jehn-Ruey Jiang. 2008. Voronoi state management for peer-to-peer massively multiplayer online games. In Proceedings of the Consumer Communications and Networking Conference (CCNC’08). 1134--1138. Google Scholar
Cross Ref
- Shun-Yun Hu, Jui-Fa Chen, and Tsu-Han Chen. 2006. VON: A scalable peer-to-peer network for virtual environments. IEEE Netw. 20, 4 (2006), 22--31. Google Scholar
Digital Library
- Shun-Yun Hu and Kuan-Ta Chen. 2011. Self-organizing spatial publish subscribe. In Proceedings of the 8th ACM International Conference on Autonomic computing (ICAC). ACM, 171--172. Google Scholar
Digital Library
- Takuji Iimura, Hiroaki Hazeyama, and Youki Kadobayashi. 2004. Zoned federation of game servers: A peer-to-peer approach to scalable multi-player online games. In Proceedings of the 3rd ACM SIGCOMM Workshop on Network and System Support for Games (NetGames). 116--120. Google Scholar
Digital Library
- Jared Jardine and Daniel Zappala. 2008. A hybrid architecture for massively multiplayer online games. In Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames). 60--65. Google Scholar
Digital Library
- Hanna Kavalionak, Emanuele Carlini, Laura Ricci, Alberto Montresor, and Massimo Coppola. 2015. Integrating peer-to-peer and cloud computing for massively multiuser online games. Peer-to-Peer Networking and Applications 8, 2 (2015), 301--319. Google Scholar
Cross Ref
- Björn Knutsson, Honghui Lu, Wei Xu, and Bryan Hopkins. 2004. Peer-to-peer support for massively multiplayer games. In Proceedings of the 23rd Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM), Vol. 1. 107. Google Scholar
Cross Ref
- Santosh Kulkarni. 2009. Badumna network suite: A decentralized network engine for massively multiplayer online applications. In Proceedings of the IEEE 9th International Conference on Peer-to-Peer Computing (P2P). 178--183. Google Scholar
Cross Ref
- Meirong Liu, Erkki Harjula, and Mika Ylianttila. 2013. An efficient selection algorithm for building a super-peer overlay. Journal of Internet Services and Applications 4, 1 (2013), 4.Google Scholar
Cross Ref
- Eng Keong Lua, J. Crowcroft, M. Pias, R. Sharma, and S. Lim. 2005. A survey and comparison of peer-to-peer overlay network schemes. IEEE Communications Surveys Tutorials 7 (2nd quarter 2005), 72--93.Google Scholar
- Martin Mauve, Jürgen Vogel, Volker Hilt, and Wolfgang Effelsberg. 2004. Local-lag and timewarp: Providing consistency for replicated continuous applications. IEEE Trans. on Multimedia 6, 1 (Feb 2004), 47--57. Google Scholar
Digital Library
- Petar Maymounkov and David Mazières. 2002. Kademlia: A peer-to-peer information system based on the XOR metric. In Revised Papers from the 1st International Workshop on Peer-to-Peer Systems (IPTPS) (IPTPS’01). Springer-Verlag, London, UK, UK, 53--65. Google Scholar
Cross Ref
- Madjid Merabti and Abdennour El Rhalibi. 2004. Peer-to-peer architecture and protocol for a massively multiplayer online game. In GlobeCom Workshops. 519--528. Google Scholar
Cross Ref
- Alberto Montresor. 2004. A robust protocol for building superpeer overlay topologies. In Proceedings of the 4th International Conference on Peer-to-Peer Computing. IEEE, Zurich, Switzerland, 202--209. Google Scholar
Cross Ref
- Alberto Montresor and Jelasity Márk. 2009. PeerSim: A scalable P2P simulator. In Proceedings of the 2009 IEEE 9th International Conference on Peer-to-Peer Computing. 99--100. Google Scholar
Cross Ref
- Matteo Mordacchini, Laura Ricci, Luca Ferrucci, Michele Albano, and Ranieri Baraglia. 2010. Hivory: Range queries on hierarchical voronoi overlays. In Proceedings of the 2010 IEEE 10th International Conference on Peer-to-Peer Computing (P2P). 1--10. Google Scholar
Cross Ref
- Marvin Rausand and Arnljot Høyland. 2004. System Reliability Theory: Models, Statistical Methods, and Applications. Wiley-Interscience.Google Scholar
- Antony Rawstron and Peter Druschel. 2001. Pastry: Scalable, decentralized object location, and routing for large-scale peer-to-peer systems. In Proceedings of the 18th IFIP/ACM International Conference on Distributed Systems Platforms (Middleware). 329--350. Google Scholar
Cross Ref
- Laura Ricci and Luca Genovali. 2010. State management in distributed virtual environments: A voronoi base approach. In Proceedings of the International Congress on Ultra Modern Telecommunications and Control Systems. 881--887. Google Scholar
Cross Ref
- Sean Rooney, Daniel Bauer, and Rudy Deydier. 2004. A federated peer-to-peer network game architecture. IEEE Commun. Mag. 42, 5 (2004), 114--122. Google Scholar
Digital Library
- Antony Rowstron and Peter Druschel. 2001. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility. ACM SIGOPS Operat. Syst. Rev. 35, 5 (2001), 188--201. Google Scholar
Digital Library
- Sebastian Schuster and Torben Weis. 2011. Enforcing game rules in untrusted P2P-based MMVEs. In Proceedings of the 4th International ICST Conference on Simulation Tools and Techniques (SIMUTools’11). ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), ICST, Brussels, Belgium, 288--295.Google Scholar
Digital Library
- Nathan Sheldon, Eric Girard, Seth Borg, Mark Claypool, and Emmanuel Agu. 2003. The effect of latency on user performance in Warcraft III. In Proceedings of the 2nd Workshop on Network and System Support for Games (NetGames’03). 3--14. Google Scholar
Digital Library
- Moritz Steiner, Taoufik En-Najjary, and Ernst W. Biersack. 2007. A global view of KAD. In Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement (IMC’07). ACM, New York, NY, 117--122. Google Scholar
Digital Library
- Ion Stoica, Robert Morris, David Karger, M. Frans Kaashoek, and Hari Balakrishnan. 2001. Chord: A scalable peer-to-peer lookup service for internet applications. SIGCOMM Comput. Commun. Rev. 31, 4 (2001), 149--160. Google Scholar
Digital Library
- Pin-Yun Tarng, Kuan-Ta Chen, and Polly Huang. 2008. An analysis of WoW players’ game hours. In Proceedings of the 7th ACM SIGCOMM Workshop on Network and System Support for Games (NetGames’08). ACM, 47--52. Google Scholar
Digital Library
- Amin Vahdat, Ken Yocum, Kevin Walsh, Priya Mahadevan, Dejan Kostić, Jeff Chase, and David Becker. 2002. Scalability and accuracy in a large-scale network emulator. ACM SIGOPS Operat. Syst. Rev. 36, SI (Dec. 2002), 271--284.Google Scholar
Digital Library
- András Varga. 2012. OMNeT++. Retrieved from http://www.omnetpp.org/.Google Scholar
- Matteo Varvello. 2009. A Peer-to-Peer Architecture for Networked Virtual Environments. Ph.D. Dissertation. Telecom Paristech.Google Scholar
- Matteo Varvello, Christophe Diot, and Ernst W. Biersack. 2009a. P2P second life: Experimental validation using kad. In Proceedings of the IEEE International Conference on Computer Communications (INFOCOM’09). 1161--1169.Google Scholar
- Matteo Varvello, Christophe Diot, and Ernst W. Biersack. 2009b. A walkable kademlia network for virtual worlds. In Proceedings of the IEEE International Conference on Computer Communications Workshops (INFOCOM’09). 1--2. Google Scholar
Cross Ref
- Matteo Varvello, Fabio Picconi, Christophe Diot, and Ernst Biersack. 2008. Is there life in second life?. In Proceedings of the 2008 ACM CoNEXT Conference (CoNEXT’08). ACM, New York, NY. Google Scholar
Digital Library
- VAST Development Team. 2010. VAST. Retrieved from vast.sourceforge.netGoogle Scholar
- Jared Winick and Sugih Jamin. 2002. Init-3.0: Internet Topology Generator. Technical Report CSE-TR-456-02. University of Mishigan.Google Scholar
- Amir Yahyavi and Bettina Kemme. 2013. Peer-to-peer architectures for massively multiplayer online games: A survey. ACM Comput. Surv. 46, 1 (July 2013), Article 9, 51 pages.Google Scholar
Digital Library
- Anthony (Peiqun) Yu and Son T. Vuong. 2005. MOPAR: A mobile peer-to-peer overlay architecture for interest management of massively multiplayer online games. In Proceedings of the International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV’05). 99--104.Google Scholar
- Jane Yang Yu and Peter H. J. Chong. 2005. A survey of clustering schemes for mobile ad hoc networks. IEEE Commun. Surv. Tutor. 7 (1st quarter 2005), 32--48.Google Scholar
- Kaiwen Zhang and Bettina Kemme. 2011. Transaction models for massively multiplayer online games. In Proceedings of the 2011 IEEE 30th International Symposium on Reliable Distributed Systems. 31--40. Google Scholar
Digital Library
Index Terms
Pithos: Distributed Storage for Massive Multi-User Virtual Environments
Recommendations
GROUP: Dual-Overlay State Management for P2P NVE
ICPADS '08: Proceedings of the 2008 14th IEEE International Conference on Parallel and Distributed SystemsPeer-to-peer (P2P) architectures have recently become a popular design choice to build scalable Networked Virtual Environments (NVEs). While P2P architectures offer better scalability than server-based architectures, efficient distribution and ...
VoroGame: a hybrid P2P architecture for massively multiplayer games
CCNC'09: Proceedings of the 6th IEEE Conference on Consumer Communications and Networking ConferencePeer-to-peer (P2P) architectures have recently become very popular in massively multiplayer games (MMGs). While P2P gaming offers high scalability compared to client/server architectures, it introduces several major issues related to data distribution ...
Reputation and audits for self-organizing storage
SOSOC '08: Proceedings of the workshop on Security in Opportunistic and SOCial networksReputation systems have demonstrated their interest in stimulating cooperation in peer-to-peer (P2P) systems. Their key operation relies on collecting, processing, and disseminating the feedback about some peers' past behavior in order to boost their ...






Comments