skip to main content
research-article

Life-Tags: A Smartglasses-based System for Recording and Abstracting Life with Tag Clouds

Published:13 June 2019Publication History
Skip Abstract Section

Abstract

We introduce "Life-Tags," a wearable, smartglasses-based system for abstracting life in the form of clouds of tags and concepts automatically extracted from snapshots of the visual reality recorded by wearable video cameras. Life-Tags summarizes users' life experiences using word clouds, highlighting the "executive summary" of what the visual experience felt like for the smartglasses user during some period of time, such as a specific day, week, month, or the last hour. In this paper, we focus on (i) design criteria and principles of operation for Life-Tags, such as its first-person, eye-level perspective for recording life, passive logging mode, and privacy-oriented operation, as well as on (ii) technical and engineering aspects for implementing Life-Tags, such as the block architecture diagram highlighting devices, software modules, third-party services, and dataflows. We also conduct a technical evaluation of Life-Tags and report results from a controlled experiment that generated 21,600 full HD snapshots from six indoor and outdoor scenarios, representative of everyday life activities, such as walking, eating, traveling, etc., with a total of 180 minutes of recorded life to abstract with tag clouds. Our experimental results and Life-Tags prototype inform design and engineering of future life abstracting systems based on smartglasses and wearable video cameras to ensure effective generation of rich clouds of concepts, reflective of the visual experience of the smartglasses user.

References

  1. R. Agarwal, N. Ladha, M. Agarwal, K. K. Majee, A. Das, S. Kumar, S. K. Rai, A. K. Singh, S. Nayak, S. Dey, R. Dey, and H. N. Saha. 2017. Low cost ultrasonic smart glasses for blind. In Proceedings of the 2017 8th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) . 210--213.Google ScholarGoogle Scholar
  2. Adrian Aiorduachioae, Radu-Daniel Vatavu, and Dorin-Mircea Popovici. 2019. A Design Space for Vehicular LifeLogging to Support Creation of Digital Content in Connected Cars. In Proceedings of the 11th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '19).Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Abdullah Al Mahmud, Jeffrey Braun, and Jean-Bernard Martens. 2010. Designing to Capture and Share Life Experiences for Persons with Aphasia. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '10). ACM, New York, NY, USA, 391--392. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Amazon. 2019. Amazon Rekognition - Video and Image - AWS. https://aws.amazon.com/rekognition/ Retrieved April 15, 2019 fromGoogle ScholarGoogle Scholar
  5. Olivier Augereau, Charles Lima Sanches, Koichi Kise, and Kai Kunze. 2018. Wordometer Systems for Everyday Life. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 1, 4, Article 123 (Jan. 2018), bibinfonumpages21 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. AYLIEN. 2019. Image Tagging | AYLIEN. https://aylien.com/text-api/image-tagging/ Retrieved April 15, 2019 fromGoogle ScholarGoogle Scholar
  7. Michael Barney, Jonathan Kilner, Gilmar Brito, Aida Araújo, and Meuse Nogueira. 2017. Sensory Glasses for the Visually Impaired. In Proceedings of the 14th Web for All Conference on The Future of Accessible Work (W4A '17). ACM, New York, NY, USA, Article 27, bibinfonumpages2 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. E. Berry, N. Kapur, L. Williams, S. Hodges, P. Watson, G. Smyth, J. Srinivasan, R. Smith, B. Wilson, and K. Wood. 2007. The use of a wearable camera, SenseCam, as a pictorial diary to improve autobiographical memory in a patient with limbic encephalitis: A preliminary report. Neuropsychol. Rehabil., Vol. 17, 4--5 (Aug-Oct 2007), 582--601.Google ScholarGoogle ScholarCross RefCross Ref
  9. Mark Blum, Alex (Sandy) Pentland, and Gerhard Troster. 2006. InSense: Interest-Based Life Logging. IEEE MultiMedia, Vol. 13, 4 (Oct. 2006), 40--48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Tianlang Chen, Yuxiao Chen, and Jiebo Luo. 2017. A Selfie is Worth a Thousand Words: Mining Personal Patterns Behind User Selfie-posting Behaviours. In Proceedings of the 26th International Conference on World Wide Web Companion (WWW '17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 23--31. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Soumyadeb Chowdhury, Md Sadek Ferdous, and Joemon M Jose. 2016. Bystander Privacy in Lifelogging. In Proceedings of the 30th International BCS Human Computer Interaction Conference: Companion Volume (HCI '16). BCS Learning & Development Ltd., Swindon, UK, Article 15, bibinfonumpages3 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Clarifai. 2019. About Clarifai. Transforming how we see the world. https://clarifai.com/about Retrieved January 25, 2019 fromGoogle ScholarGoogle Scholar
  13. Google Clips. 2019. Google Clips Specifications. https://support.google.com/googleclips/answer/7545447?hl=en Retrieved February 12, 2019 fromGoogle ScholarGoogle Scholar
  14. Will Coldwell. 2014. Autographer wearable camera: reviewed on a Berlin city break. https://www.theguardian.com/travel/2014/feb/27/autographer-wearable-camera-review-berlin-city-break Retrieved February 12, 2019 fromGoogle ScholarGoogle Scholar
  15. Dirk de Jager, Alex L. Wood, Geoff V. Merrett, Bashir M. Al-Hashimi, Kieron O'Hara, Nigel R. Shadbolt, and Wendy Hall. 2011. A Low-power, Distributed, Pervasive Healthcare System for Supporting Memory. In Proceedings of the First ACM MobiHoc Workshop on Pervasive Wireless Healthcare (MobileHealth '11). ACM, New York, NY, USA, Article 5, bibinfonumpages7 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Yuan Ding, Yuan Du, Yingkai Hu, Zhengye Liu, Luqin Wang, Keith Ross, and Anindya Ghose. 2011. Broadcast Yourself: Understanding YouTube Uploaders. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference (IMC '11). ACM, New York, NY, USA, 361--370. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Martin Dodge and Rob Kitchin. 2007. 'Outlines of a World Coming into Existence': Pervasive Computing and the Ethics of Forgetting. Environment and Planning B: Planning and Design, Vol. 34, 3 (2007), 431--445.Google ScholarGoogle Scholar
  18. Gartner. 2018. Gartner says worldwide wearable device sales to grow 26 percent in 2019. https://www.gartner.com/en/newsroom/press-releases/2018--11--29-gartner-says-worldwide-wearable-device-sales-to-grow- Retrieved February 12, 2019 fromGoogle ScholarGoogle Scholar
  19. Jim Gemmell. 2014. Life-logging, Thing-logging and the Internet of Things. In Proceedings of the 2014 Workshop on Physical Analytics (WPA '14). ACM, New York, NY, USA, 17--17. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Bogdan-Florin Gheran, Jean Vanderdonckt, and Radu-Daniel Vatavu. 2018. Gestures for Smart Rings: Empirical Results, Insights, and Design Implications. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 623--635.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Google. 2019. Vision AI. Derive Image Insights with ML. Google Cloud. https://cloud.google.com/vision/ Retrieved April 15, 2019 fromGoogle ScholarGoogle Scholar
  22. Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging: Personal Big Data. Found. Trends Inf. Retr., Vol. 8, 1 (June 2014), 1--125. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Phil Hall. 2019. Best action camera 2019: 10 cameras for the GoPro generation. https://www.techradar.com/news/best-action-camera Retrieved February 12, 2019 fromGoogle ScholarGoogle Scholar
  24. Hugh Hart. 2010. April 14, 1996: JenniCam starts lifecasting. https://www.wired.com/2010/04/0414jennicam-launches/ Retrieved April 14, 2019 fromGoogle ScholarGoogle Scholar
  25. Julia Heidemann, Mathias Klier, and Florian Probst. 2012. Online social networks: A survey of a global phenomenon. Computer Networks, Vol. 56, 18 (2012), 3866--3878. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. SenseCam: A Retrospective Memory Aid. In Proceedings of the 8th International Conference on Ubiquitous Computing (UbiComp'06). Springer-Verlag, Berlin, Heidelberg, 177--193.Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014. Privacy Behaviors of Lifeloggers Using Wearable Cameras. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). ACM, New York, NY, USA, 571--582. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Chunjia Hu, Guangtao Zhai, and Duo Li. 2015. An Augmented-Reality night vision enhancement application for see-through glasses. In 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW) . 1--6.Google ScholarGoogle Scholar
  29. Rui-Ting Huang. 2018. What Motivates People to Continuously Post Selfies? The Moderating Role of Perceived Relative Advantage. Comput. Hum. Behav., Vol. 80, C (March 2018), 103--111. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Mohammad Hossein Jarrahi, Nicci Gafinowitz, and Grace Shin. 2018. Activity Trackers, Prior Motivation, and Perceived Informational and Motivational Affordances. Personal Ubiquitous Comput., Vol. 22, 2 (April 2018), 433--448. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Hayeon Jeong, Heepyung Kim, Rihun Kim, Uichin Lee, and Yong Jeong. 2017. Smartwatch Wearing Behavior Analysis: A Longitudinal Study. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 1, 3, Article 60 (Sept. 2017), bibinfonumpages31 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct). ACM, New York, NY, USA, 1151--1160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Keigo Kitamura, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2008. Food Log by Analyzing Food Images. In Proceedings of the 16th ACM International Conference on Multimedia (MM '08). ACM, New York, NY, USA, 999--1000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Marion Koelle, Matthias Kranz, and Andreas Möller. 2015. Don't Look at Me That Way!: Understanding User Attitudes Towards Data Glasses Usage. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, New York, NY, USA, 362--372.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Hyowon Lee, Alan F. Smeaton, Noel E. O'connor, Gareth Jones, Michael Blighe, Daragh Byrne, Aiden Doherty, and Cathal Gurrin. 2008. Constructing a SenseCam Visual Diary As a Media Process. Multimedia Syst., Vol. 14, 6 (Dec. 2008), 341--349. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Andrés Lucero. 2018. Living Without a Mobile Phone: An Autoethnography. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 765--776.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. J. Machajdik, A. Hanbury, A. Garz, and R. Sablatnig. 2011. Affective Computing for Wearable Diary and Lifelogging Systems: An Overview. In Proceedings of the 35th Workshop of the Austrian Association for Pattern Recognition . Austrian Computer Society. https://www.ec.tuwien.ac.at/node/6421Google ScholarGoogle Scholar
  38. Matei Mancas, Vincent P. Ferrera, Nicolas Riche, and John G. Taylor. 2016. From Human Attention to Computational Attention: A Multidisciplinary Approach 1st ed.). Springer Publishing Company, Inc. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. Steve Mann, James Fung, and Eric Moncrieff. 1999. EyeTap Technology for Wireless Electronic News Gathering. SIGMOBILE Mob. Comput. Commun. Rev., Vol. 3, 4 (Oct. 1999), 19--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Joshua McVeigh-Schultz, Jennifer Stein, Jacob Boyle, Emily Duff, Jeff Watson, Avimaan Syam, Amanda Tasse, Simon Wiscombe, and Scott Fisher. 2012. Vehicular Lifelogging: New Contexts and Methodologies for Human-car Interaction. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 221--230. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. MeCam. 2019. High definition video camera | Best life logging device | Mini video camera - MeCam. https://mecam.me/products/mecam-hd Retrieved February 12, 2019 fromGoogle ScholarGoogle Scholar
  42. Vangelis Metsis, Dimitrios Kosmopoulos, Vassilis Athitsos, and Fillia Makedon. 2014. Non-invasive Analysis of Sleep Patterns via Multimodal Sensor Input. Personal and Ubiquitous Computing, Vol. 18, 1 (Jan. 2014), 19--26. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Narrative. 2019. The World's Most Wearable HD Video Camera | Narrative Clip 2. http://getnarrative.com/ Retrieved February 12, 2019 fromGoogle ScholarGoogle Scholar
  44. Donald A. Norman. 2004. Emotional Design. Why We Love (or Hate) Everyday Things .Basic Books, New York, NY, USA. https://www.nngroup.com/books/emotional-design/Google ScholarGoogle Scholar
  45. Bogdan Pogorelc, Artur Lugmayr, Bjorn Stockleben, Radu-Daniel Vatavu, Nina Tahmasebi, Estefania Serral, Emilija Stojmenova, Bojan Imperl, Thomas Risse, Gideon Zenz, and Matjaz Gams. 2013. Ambient Bloom: New Business, Content, Design and Models to Increase the Semantic Ambient Media Experience. Multimedia Tools and Applications, Vol. 66, 1 (2013), 7--32. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Andrei Popleteev, Nicolas Louveton, and Roderick McCall. 2015. Colorizer: Smart Glasses Aid for the Colorblind. In Proceedings of the 2015 Workshop on Wearable Systems and Applications (WearSys '15). ACM, New York, NY, USA, 7--8.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. RedShed. 2018. The 8 Best Spy Glasses of 2018. https://redshed.co.uk/tech-reviews/best-spy-glasses/ Retrieved January 25, 2019 fromGoogle ScholarGoogle Scholar
  48. Ovidiu-Andrei Schipor, Radu-Daniel Vatavu, and Jean Vanderdonckt. 2019. Euphoria: A Scalable, Event-Driven Architecture for Designing Interactions Across Heterogeneous Devices in Smart Environments. Information and Software Technology, Vol. 109 (2019), 43--59.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Courtney Seiter. 2016. The Psychology of Social Media: Why We Like, Comment, and Share Online. https://blog.bufferapp.com/psychology-of-social-media Retrieved January 25, 2019 fromGoogle ScholarGoogle Scholar
  50. Abigail J. Sellen and Steve Whittaker. 2010. Beyond Total Capture: A Constructive Critique of Lifelogging. Commun. ACM, Vol. 53, 5 (May 2010), 70--77. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. Kristen Shinohara and Josh Tenenberg. 2007. Observing Sara: A Case Study of a Blind Person's Interactions with Technology. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '07). ACM, New York, NY, USA, 171--178. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Snap. 2019. Spectacles by uppercaseSnapchat | uppercaseYour Hands-Free Camera. https://www.spectacles.com/ Retrieved January 25, 2019 fromGoogle ScholarGoogle Scholar
  53. SnapCam. 2019. iON USA | SnapCam. https://usa.ioncamera.com/snapcam/ Retrieved February 12, 2019 fromGoogle ScholarGoogle Scholar
  54. Flávio Souza, Diego de Las Casas, Vin'icius Flores, SunBum Youn, Meeyoung Cha, Daniele Quercia, and Virg'ilio Almeida. 2015. Dawn of the Selfie Era: The Whos, Wheres, and Hows of Selfies on Instagram. In Proceedings of the 2015 ACM on Conference on Online Social Networks (COSN '15). ACM, New York, NY, USA, 221--231. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Gary Stix. 2011. Photographic Memory: Wearable Cam Could Help Patients Stave Off Effects of Impaired Recall. Scientific American (October 2011). https://www.scientificamerican.com/article/photographic-memory-wearable/Google ScholarGoogle ScholarCross RefCross Ref
  56. Radu-Daniel Vatavu. 2012. Presence bubbles: supporting and enhancing human-human interaction with ambient media. Multimedia Tools and Applications, Vol. 58, 2 (May 2012), 371--383. Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. Seung-Ho Yang, Hyun-Woo Kim, and Min Young Kim. 2011. Human Visual Augmentation Using Wearable Glasses with Multiple Cameras and Information Fusion of Human Eye Tracking and Scene Understanding. In Proceedings of the 6th International Conference on Human-robot Interaction (HRI '11). ACM, New York, NY, USA, 287--288. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Hinbarji Z., Albatal R., O'Connor N., and Gurrin C. 2016. LoggerMan, a Comprehensive Logging and Visualization Tool to Capture Computer Usage. Vol. 9517. Springer, Cham.Google ScholarGoogle Scholar
  59. Yuhang Zhao, Sarit Szpiro, and Shiri Azenkot. 2015. ForeSee: A Customizable Head-Mounted Vision Enhancement System for People with Low Vision. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '15). ACM, New York, NY, USA, 239--249. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Yuhang Zhao, Sarit Szpiro, Jonathan Knighten, and Shiri Azenkot. 2016. CueSee: Exploring Visual Cues for People with Low Vision to Facilitate a Visual Search Task. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 73--84.Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. Liting Zhou, Zaher Hinbarji, Duc-Tien Dang-Nguyen, and Cathal Gurrin. 2018. LIFER: An Interactive Lifelog Retrieval System. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge (LSC '18). ACM, New York, NY, USA, 9--14.Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Floriano Zini, Martin Reinstadler, and Francesco Ricci. 2015. Life-logs Aggregation for Quality of Life Monitoring. In Proceedings of the 5th International Conference on Digital Health 2015 (DH '15). ACM, New York, NY, USA, 131--132. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Life-Tags: A Smartglasses-based System for Recording and Abstracting Life with Tag Clouds

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!