Creepy Tracker Toolkit for Context-aware Interfaces

ABSTRACT

Context-aware pervasive applications can improve user experiences by tracking people in their surroundings. Such systems use multiple sensors to gather information regarding people and devices. However, when developing novel user experiences, researchers are left to building foundation code to support multiple network-connected sensors, a major hurdle to rapidly developing and testing new ideas.

We introduce Creepy Tracker, an open-source toolkit to ease prototyping with multiple commodity depth cameras. It automatically selects the best sensor to follow each person, handling occlusions and maximizing interaction space, while providing full-body tracking in scalable and extensible manners. It also keeps position and orientation of stationary interactive surfaces while offering continuously updated point-cloud user representations combining both depth and color data. Our performance evaluation shows that, although slightly less precise than marker-based optical systems, Creepy Tracker provides reliable multi-joint tracking without any wearable markers or special devices. Furthermore, implemented representative scenarios show that Creepy Tracker is well suited for deploying spatial and context-aware interactive experiences.

References

1. Aggarwal, J. K., and Ryoo, M. S. Human activity analysis: A review. ACM Computing Surveys (CSUR) 43, 3 (2011), 16.
2. Annett, M., Grossman, T., Wigdor, D., and Fitzmaurice, G. Medusa: a proximity-aware multi-touch tabletop. In Proceedings of the 24th annual ACM symposium on User interface software and technology, ACM (2011), 337--346.
3. Antifakos, S., and Schiele, B. Beyond position awareness. Personal and Ubiquitous Computing 6, 5--6 (2002), 313--317.
4. Ark, W. S., and Selker, T. A look at human interaction with pervasive computers. IBM systems journal 38, 4 (1999), 504--507.
5. Augsten, T., Kaefer, K., Meusel, R., Fetzer, C., Kanitz, D., Stoff, T., Becker, T., Holz, C., and Baudisch, P. Multitoe: High-precision interaction with back-projected floors based on high-resolution multi-touch input. In Proceedings of the 23Nd Annual ACM Symposium on User Interface Software and Technology, UIST '10, ACM (New York, NY, USA, 2010), 209--218.
6. Azimi, M. Skeletal joint smoothing white paper. Tech. rep., 2012. http://msdn.microsoft.com/en-us/library/jj131429.aspx.
7. Ballendat, T., Marquardt, N., and Greenberg, S. Proxemic interaction: Designing for a proximity and orientation-aware environment. In ACM International Conference on Interactive Tabletops and Surfaces, ITS '10, ACM (New York, NY, USA, 2010), 121--130.
8. Beck, S., Kunert, A., Kulik, A., and Froehlich, B. Immersive group-to-group telepresence. IEEE Transactions on Visualization and Computer Graphics 19, 4 (2013), 616--625.
9. Bolt, R. A. Put-that-there: Voice and gesture at the graphics interface. In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '80, ACM (New York, NY, USA, 1980), 262--270.
10. Dey, A. K., Abowd, G. D., and Salber, D. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-computer interaction 16, 2 (2001), 97--166.
11. Fails, J. A., and Olsen, J. D. Light widgets: interacting in every-day spaces. In Proceedings of the 7th international conference on Intelligent user interfaces, ACM (2002), 63--69.
12. Grønbæk, K., Iversen, O. S., Kortbek, K. J., Nielsen, K. R., and Aagaard, L. Igamefloor: A platform for co-located collaborative games. In Proceedings of the International Conference on Advances in Computer Entertainment Technology, ACE '07, ACM (New York, NY, USA, 2007), 64--71.
13. Gugenheimer, J., Stemasov, E., Frommel, J., and Rukzio, E. Sharevr: Enabling co-located experiences for virtual reality between hmd and non-hmd users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, ACM (New York, NY, USA, 2017), 4021--1033.
14. Hall, E. T. The hidden dimension. Doubleday & Co, 1966.
15. Harris, A., Rick, J., Bonnett, V., Yuill, N., Fleck, R., Marshall, P., and Rogers, Y. Around the table: Are multiple-touch surfaces better than single-touch for children's collaborative interactions? In Proceedings of the 9th international conference on Computer supported collaborative learning-Volume 1, International Society of the Learning Sciences (2009), 335--344.
16. Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. arXiv preprint arXiv:1605.03170 (2016).
17. Jokela, T., Ojala, J., and Olsson, T. A diary study on combining multiple information devices in everyday activities and tasks. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, ACM (New York, NY, USA, 2015), 3903--3912.
18. Jones, B., Sodhi, R., Murdock, M., Mehra, R., Benko, H., Wilson, A., Ofek, E., MacIntyre, B., Raghuvanshi, N., and Shapira, L. Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology, UIST '14, ACM (New York, NY, USA, 2014), 637--644.
19. Marquardt, N., Ballendat, T., Boring, S., Greenberg, S., and Hinckley, K. Gradual engagement: Facilitating information exchange between digital devices as a function of proximity. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces, ITS '12, ACM (New York, NY, USA, 2012), 31--40.
20. Marquardt, N., Diaz-Marino, R., Boring, S., and Greenberg, S. The proximity toolkit: Prototyping proxemic interactions in ubiquitous computing ecologies. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST '11, ACM (New York, NY, USA, 2011), 315--326.
21. Marquardt, N., Hinckley, K., and Greenberg, S. Cross-device interaction via micro-mobility and f-formations. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST '12, ACM (New York, NY, USA, 2012), 13--22.
22. Marquardt, N., Jota, R., Greenberg, S., and Jorge, J. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. Human-Computer Interaction--INTERACT 2011 (2011), 461--476.
23. Marshall, P., Hornecker, E., Morris, R., Dalton, N. S., and Rogers, Y. When the fingers do the talking: A study of group participation with varying constraints to a tabletop interface. In Horizontal Interactive Human Computer Systems, 2008. TABLETOP 2008. 3rd IEEE International Workshop on, IEEE (2008), 33--10.
24. Matthews, T., Dey, A. K., Mankoff, J., Carter, S., and Rattenbury, T. A toolkit for managing user attention in peripheral displays. In Proceedings of the 17th annual ACM symposium on User interface software and technology, ACM (2004), 247--256.
25. Mendes, D., Relvas, F., Ferreira, A., and Jorge, J. The benefits of dof separation in mid-air 3d object manipulation. In Proceedings of the 22Nd ACM Conference on Virtual Reality Software and Technology, VRST '16, ACM (New York, NY, USA, 2016), 261--268.
26. Orts-Escolano, S., Rhemann, C., Fanello, S., Chang, W., Kowdle, A., Degtyarev, Y., Kim, D., Davidson, P. L., Khamis, S., Dou, M., et al. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, ACM (2016), 741--754.
27. Pejsa, T., Kantor, J., Benko, H., Ofek, E., and Wilson, A. Room2room: Enabling life-size telepresence in a projected augmented reality environment. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing, ACM (2016), 1716--1725.
28. Pierce, J. S., Forsberg, A. S., Conway, M. J., Hong, S., Zeleznik, R. C., and Mine, M. R. Image plane interaction techniques in 3d immersive environments. In Proceedings of the 1997 symposium on Interactive 3D graphics, ACM (1997), 39-ff.
29. Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '98, ACM (New York, NY, USA, 1998), 179--188.
30. Rekimoto, J., and Nagao, K. The world through the computer: Computer augmented interaction with real world environments. In Proceedings of the 8th annual ACM symposium on User interface and software technology, ACM (1995), 29--36.
31. Schilit, B., Adams, N., and Want, R. Context-aware computing applications. In Mobile Computing Systems and Applications, 1994. WMCSA 1994. First Workshop on, IEEE (1994), 85--90.
32. Seyed, T., Azazi, A., Chan, E., Wang, Y., and Maurer, F. Sod-toolkit: A toolkit for interactively prototyping and developing multi-sensor, multi-device environments. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces, ITS '15, ACM (New York, NY, USA, 2015), 171--180.
33. Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Cook, M., and Moore, R. Real-time human pose recognition in parts from single depth images. Communications of the ACM 56, 1 (2013), 116--124.
34. Slater, M., and Usoh, M. Body centred interaction in immersive virtual environments. Artificial life and virtual reality 1, 1994 (1994), 125--148.
35. Sousa, M., Mendes, D., Ferreira, A., Pereira, J. M., and Jorge, J. Eery space: facilitating virtual meetings through remote proxemics. In Human-Computer Interaction, Springer, Cham (2015), 622--629.
36. Tang, R., Alizadeh, H., Tang, A., Bateman, S., and Jorge, J. A. [email protected]: Design explorations to support movement guidance. In CHI '14 Extended Abstracts on Human Factors in Computing Systems, CHI EA '14, ACM (New York, NY, USA, 2014), 1651--1656.
37. Vermeulen, J., Luyten, K., Coninx, K., Marquardt, N., and Bird, J. Proxemic flow: Dynamic peripheral floor visualizations for revealing and mediating large surface interactions. In Human-Computer Interaction, Springer (2015), 264--281.
38. Vogel, D., and Balakrishnan, R. Interactive public ambient displays: Transitioning from implicit to explicit, public to personal, interaction with multiple users. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, UIST '04, ACM (New York, NY, USA, 2004), 137--146.
39. Weiser, M. The computer for the 21st century. Scientific american 265, 3 (1991), 94--104.
40. Weiser, M. The computer for the 21st century. IEEE pervasive computing 1, 1 (2002), 19--25.
41. Wilson, A. D. Depth-sensing video cameras for 3d tangible tabletop interaction. In Horizontal Interactive Human-Computer Systems, 2007. TABLETOP'07. Second Annual IEEE International Workshop on, IEEE (2007), 201--204.
42. Wilson, A. D., and Benko, H. Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM (2010), 273--282.
43. Wilson, A. D., and Benko, H. Crossmotion: Fusing device and image motion for user identification, tracking and device association. In Proceedings of the 16th International Conference on Multimodal Interaction, ICMI '14, ACM (New York, NY, USA, 2014), 216--223.
44. Wu, C.-J., Houben, S., and Marquardt, N. Eaglesense: Tracking people and devices in interactive spaces using real-time top-view depth-sensing. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, ACM (New York, NY, USA, 2017), 3929--3942.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)