Abstract
We introduce "Life-Tags," a wearable, smartglasses-based system for abstracting life in the form of clouds of tags and concepts automatically extracted from snapshots of the visual reality recorded by wearable video cameras. Life-Tags summarizes users' life experiences using word clouds, highlighting the "executive summary" of what the visual experience felt like for the smartglasses user during some period of time, such as a specific day, week, month, or the last hour. In this paper, we focus on (i) design criteria and principles of operation for Life-Tags, such as its first-person, eye-level perspective for recording life, passive logging mode, and privacy-oriented operation, as well as on (ii) technical and engineering aspects for implementing Life-Tags, such as the block architecture diagram highlighting devices, software modules, third-party services, and dataflows. We also conduct a technical evaluation of Life-Tags and report results from a controlled experiment that generated 21,600 full HD snapshots from six indoor and outdoor scenarios, representative of everyday life activities, such as walking, eating, traveling, etc., with a total of 180 minutes of recorded life to abstract with tag clouds. Our experimental results and Life-Tags prototype inform design and engineering of future life abstracting systems based on smartglasses and wearable video cameras to ensure effective generation of rich clouds of concepts, reflective of the visual experience of the smartglasses user.
- R. Agarwal, N. Ladha, M. Agarwal, K. K. Majee, A. Das, S. Kumar, S. K. Rai, A. K. Singh, S. Nayak, S. Dey, R. Dey, and H. N. Saha. 2017. Low cost ultrasonic smart glasses for blind. In Proceedings of the 2017 8th IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) . 210--213.Google Scholar
- Adrian Aiorduachioae, Radu-Daniel Vatavu, and Dorin-Mircea Popovici. 2019. A Design Space for Vehicular LifeLogging to Support Creation of Digital Content in Connected Cars. In Proceedings of the 11th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '19).Google Scholar
Digital Library
- Abdullah Al Mahmud, Jeffrey Braun, and Jean-Bernard Martens. 2010. Designing to Capture and Share Life Experiences for Persons with Aphasia. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '10). ACM, New York, NY, USA, 391--392. Google Scholar
Digital Library
- Amazon. 2019. Amazon Rekognition - Video and Image - AWS. https://aws.amazon.com/rekognition/ Retrieved April 15, 2019 fromGoogle Scholar
- Olivier Augereau, Charles Lima Sanches, Koichi Kise, and Kai Kunze. 2018. Wordometer Systems for Everyday Life. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 1, 4, Article 123 (Jan. 2018), bibinfonumpages21 pages.Google Scholar
Digital Library
- AYLIEN. 2019. Image Tagging | AYLIEN. https://aylien.com/text-api/image-tagging/ Retrieved April 15, 2019 fromGoogle Scholar
- Michael Barney, Jonathan Kilner, Gilmar Brito, Aida Araújo, and Meuse Nogueira. 2017. Sensory Glasses for the Visually Impaired. In Proceedings of the 14th Web for All Conference on The Future of Accessible Work (W4A '17). ACM, New York, NY, USA, Article 27, bibinfonumpages2 pages.Google Scholar
Digital Library
- E. Berry, N. Kapur, L. Williams, S. Hodges, P. Watson, G. Smyth, J. Srinivasan, R. Smith, B. Wilson, and K. Wood. 2007. The use of a wearable camera, SenseCam, as a pictorial diary to improve autobiographical memory in a patient with limbic encephalitis: A preliminary report. Neuropsychol. Rehabil., Vol. 17, 4--5 (Aug-Oct 2007), 582--601.Google Scholar
Cross Ref
- Mark Blum, Alex (Sandy) Pentland, and Gerhard Troster. 2006. InSense: Interest-Based Life Logging. IEEE MultiMedia, Vol. 13, 4 (Oct. 2006), 40--48. Google Scholar
Digital Library
- Tianlang Chen, Yuxiao Chen, and Jiebo Luo. 2017. A Selfie is Worth a Thousand Words: Mining Personal Patterns Behind User Selfie-posting Behaviours. In Proceedings of the 26th International Conference on World Wide Web Companion (WWW '17 Companion). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 23--31. Google Scholar
Digital Library
- Soumyadeb Chowdhury, Md Sadek Ferdous, and Joemon M Jose. 2016. Bystander Privacy in Lifelogging. In Proceedings of the 30th International BCS Human Computer Interaction Conference: Companion Volume (HCI '16). BCS Learning & Development Ltd., Swindon, UK, Article 15, bibinfonumpages3 pages. Google Scholar
Digital Library
- Clarifai. 2019. About Clarifai. Transforming how we see the world. https://clarifai.com/about Retrieved January 25, 2019 fromGoogle Scholar
- Google Clips. 2019. Google Clips Specifications. https://support.google.com/googleclips/answer/7545447?hl=en Retrieved February 12, 2019 fromGoogle Scholar
- Will Coldwell. 2014. Autographer wearable camera: reviewed on a Berlin city break. https://www.theguardian.com/travel/2014/feb/27/autographer-wearable-camera-review-berlin-city-break Retrieved February 12, 2019 fromGoogle Scholar
- Dirk de Jager, Alex L. Wood, Geoff V. Merrett, Bashir M. Al-Hashimi, Kieron O'Hara, Nigel R. Shadbolt, and Wendy Hall. 2011. A Low-power, Distributed, Pervasive Healthcare System for Supporting Memory. In Proceedings of the First ACM MobiHoc Workshop on Pervasive Wireless Healthcare (MobileHealth '11). ACM, New York, NY, USA, Article 5, bibinfonumpages7 pages. Google Scholar
Digital Library
- Yuan Ding, Yuan Du, Yingkai Hu, Zhengye Liu, Luqin Wang, Keith Ross, and Anindya Ghose. 2011. Broadcast Yourself: Understanding YouTube Uploaders. In Proceedings of the 2011 ACM SIGCOMM Conference on Internet Measurement Conference (IMC '11). ACM, New York, NY, USA, 361--370. Google Scholar
Digital Library
- Martin Dodge and Rob Kitchin. 2007. 'Outlines of a World Coming into Existence': Pervasive Computing and the Ethics of Forgetting. Environment and Planning B: Planning and Design, Vol. 34, 3 (2007), 431--445.Google Scholar
- Gartner. 2018. Gartner says worldwide wearable device sales to grow 26 percent in 2019. https://www.gartner.com/en/newsroom/press-releases/2018--11--29-gartner-says-worldwide-wearable-device-sales-to-grow- Retrieved February 12, 2019 fromGoogle Scholar
- Jim Gemmell. 2014. Life-logging, Thing-logging and the Internet of Things. In Proceedings of the 2014 Workshop on Physical Analytics (WPA '14). ACM, New York, NY, USA, 17--17. Google Scholar
Digital Library
- Bogdan-Florin Gheran, Jean Vanderdonckt, and Radu-Daniel Vatavu. 2018. Gestures for Smart Rings: Empirical Results, Insights, and Design Implications. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 623--635.Google Scholar
Digital Library
- Google. 2019. Vision AI. Derive Image Insights with ML. Google Cloud. https://cloud.google.com/vision/ Retrieved April 15, 2019 fromGoogle Scholar
- Cathal Gurrin, Alan F. Smeaton, and Aiden R. Doherty. 2014. LifeLogging: Personal Big Data. Found. Trends Inf. Retr., Vol. 8, 1 (June 2014), 1--125. Google Scholar
Digital Library
- Phil Hall. 2019. Best action camera 2019: 10 cameras for the GoPro generation. https://www.techradar.com/news/best-action-camera Retrieved February 12, 2019 fromGoogle Scholar
- Hugh Hart. 2010. April 14, 1996: JenniCam starts lifecasting. https://www.wired.com/2010/04/0414jennicam-launches/ Retrieved April 14, 2019 fromGoogle Scholar
- Julia Heidemann, Mathias Klier, and Florian Probst. 2012. Online social networks: A survey of a global phenomenon. Computer Networks, Vol. 56, 18 (2012), 3866--3878. Google Scholar
Digital Library
- Steve Hodges, Lyndsay Williams, Emma Berry, Shahram Izadi, James Srinivasan, Alex Butler, Gavin Smyth, Narinder Kapur, and Ken Wood. 2006. SenseCam: A Retrospective Memory Aid. In Proceedings of the 8th International Conference on Ubiquitous Computing (UbiComp'06). Springer-Verlag, Berlin, Heidelberg, 177--193.Google Scholar
Digital Library
- Roberto Hoyle, Robert Templeman, Steven Armes, Denise Anthony, David Crandall, and Apu Kapadia. 2014. Privacy Behaviors of Lifeloggers Using Wearable Cameras. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '14). ACM, New York, NY, USA, 571--582. Google Scholar
Digital Library
- Chunjia Hu, Guangtao Zhai, and Duo Li. 2015. An Augmented-Reality night vision enhancement application for see-through glasses. In 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW) . 1--6.Google Scholar
- Rui-Ting Huang. 2018. What Motivates People to Continuously Post Selfies? The Moderating Role of Perceived Relative Advantage. Comput. Hum. Behav., Vol. 80, C (March 2018), 103--111. Google Scholar
Digital Library
- Mohammad Hossein Jarrahi, Nicci Gafinowitz, and Grace Shin. 2018. Activity Trackers, Prior Motivation, and Perceived Informational and Motivational Affordances. Personal Ubiquitous Comput., Vol. 22, 2 (April 2018), 433--448. Google Scholar
Digital Library
- Hayeon Jeong, Heepyung Kim, Rihun Kim, Uichin Lee, and Yong Jeong. 2017. Smartwatch Wearing Behavior Analysis: A Longitudinal Study. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 1, 3, Article 60 (Sept. 2017), bibinfonumpages31 pages.Google Scholar
Digital Library
- Moritz Kassner, William Patera, and Andreas Bulling. 2014. Pupil: An Open Source Platform for Pervasive Eye Tracking and Mobile Gaze-based Interaction. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication (UbiComp '14 Adjunct). ACM, New York, NY, USA, 1151--1160. Google Scholar
Digital Library
- Keigo Kitamura, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2008. Food Log by Analyzing Food Images. In Proceedings of the 16th ACM International Conference on Multimedia (MM '08). ACM, New York, NY, USA, 999--1000. Google Scholar
Digital Library
- Marion Koelle, Matthias Kranz, and Andreas Möller. 2015. Don't Look at Me That Way!: Understanding User Attitudes Towards Data Glasses Usage. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '15). ACM, New York, NY, USA, 362--372.Google Scholar
Digital Library
- Hyowon Lee, Alan F. Smeaton, Noel E. O'connor, Gareth Jones, Michael Blighe, Daragh Byrne, Aiden Doherty, and Cathal Gurrin. 2008. Constructing a SenseCam Visual Diary As a Media Process. Multimedia Syst., Vol. 14, 6 (Dec. 2008), 341--349. Google Scholar
Digital Library
- Andrés Lucero. 2018. Living Without a Mobile Phone: An Autoethnography. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 765--776.Google Scholar
Digital Library
- J. Machajdik, A. Hanbury, A. Garz, and R. Sablatnig. 2011. Affective Computing for Wearable Diary and Lifelogging Systems: An Overview. In Proceedings of the 35th Workshop of the Austrian Association for Pattern Recognition . Austrian Computer Society. https://www.ec.tuwien.ac.at/node/6421Google Scholar
- Matei Mancas, Vincent P. Ferrera, Nicolas Riche, and John G. Taylor. 2016. From Human Attention to Computational Attention: A Multidisciplinary Approach 1st ed.). Springer Publishing Company, Inc. Google Scholar
Digital Library
- Steve Mann, James Fung, and Eric Moncrieff. 1999. EyeTap Technology for Wireless Electronic News Gathering. SIGMOBILE Mob. Comput. Commun. Rev., Vol. 3, 4 (Oct. 1999), 19--26. Google Scholar
Digital Library
- Joshua McVeigh-Schultz, Jennifer Stein, Jacob Boyle, Emily Duff, Jeff Watson, Avimaan Syam, Amanda Tasse, Simon Wiscombe, and Scott Fisher. 2012. Vehicular Lifelogging: New Contexts and Methodologies for Human-car Interaction. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 221--230. Google Scholar
Digital Library
- MeCam. 2019. High definition video camera | Best life logging device | Mini video camera - MeCam. https://mecam.me/products/mecam-hd Retrieved February 12, 2019 fromGoogle Scholar
- Vangelis Metsis, Dimitrios Kosmopoulos, Vassilis Athitsos, and Fillia Makedon. 2014. Non-invasive Analysis of Sleep Patterns via Multimodal Sensor Input. Personal and Ubiquitous Computing, Vol. 18, 1 (Jan. 2014), 19--26. Google Scholar
Digital Library
- Narrative. 2019. The World's Most Wearable HD Video Camera | Narrative Clip 2. http://getnarrative.com/ Retrieved February 12, 2019 fromGoogle Scholar
- Donald A. Norman. 2004. Emotional Design. Why We Love (or Hate) Everyday Things .Basic Books, New York, NY, USA. https://www.nngroup.com/books/emotional-design/Google Scholar
- Bogdan Pogorelc, Artur Lugmayr, Bjorn Stockleben, Radu-Daniel Vatavu, Nina Tahmasebi, Estefania Serral, Emilija Stojmenova, Bojan Imperl, Thomas Risse, Gideon Zenz, and Matjaz Gams. 2013. Ambient Bloom: New Business, Content, Design and Models to Increase the Semantic Ambient Media Experience. Multimedia Tools and Applications, Vol. 66, 1 (2013), 7--32. Google Scholar
Digital Library
- Andrei Popleteev, Nicolas Louveton, and Roderick McCall. 2015. Colorizer: Smart Glasses Aid for the Colorblind. In Proceedings of the 2015 Workshop on Wearable Systems and Applications (WearSys '15). ACM, New York, NY, USA, 7--8.Google Scholar
Digital Library
- RedShed. 2018. The 8 Best Spy Glasses of 2018. https://redshed.co.uk/tech-reviews/best-spy-glasses/ Retrieved January 25, 2019 fromGoogle Scholar
- Ovidiu-Andrei Schipor, Radu-Daniel Vatavu, and Jean Vanderdonckt. 2019. Euphoria: A Scalable, Event-Driven Architecture for Designing Interactions Across Heterogeneous Devices in Smart Environments. Information and Software Technology, Vol. 109 (2019), 43--59.Google Scholar
Digital Library
- Courtney Seiter. 2016. The Psychology of Social Media: Why We Like, Comment, and Share Online. https://blog.bufferapp.com/psychology-of-social-media Retrieved January 25, 2019 fromGoogle Scholar
- Abigail J. Sellen and Steve Whittaker. 2010. Beyond Total Capture: A Constructive Critique of Lifelogging. Commun. ACM, Vol. 53, 5 (May 2010), 70--77. Google Scholar
Digital Library
- Kristen Shinohara and Josh Tenenberg. 2007. Observing Sara: A Case Study of a Blind Person's Interactions with Technology. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '07). ACM, New York, NY, USA, 171--178. Google Scholar
Digital Library
- Snap. 2019. Spectacles by uppercaseSnapchat | uppercaseYour Hands-Free Camera. https://www.spectacles.com/ Retrieved January 25, 2019 fromGoogle Scholar
- SnapCam. 2019. iON USA | SnapCam. https://usa.ioncamera.com/snapcam/ Retrieved February 12, 2019 fromGoogle Scholar
- Flávio Souza, Diego de Las Casas, Vin'icius Flores, SunBum Youn, Meeyoung Cha, Daniele Quercia, and Virg'ilio Almeida. 2015. Dawn of the Selfie Era: The Whos, Wheres, and Hows of Selfies on Instagram. In Proceedings of the 2015 ACM on Conference on Online Social Networks (COSN '15). ACM, New York, NY, USA, 221--231. Google Scholar
Digital Library
- Gary Stix. 2011. Photographic Memory: Wearable Cam Could Help Patients Stave Off Effects of Impaired Recall. Scientific American (October 2011). https://www.scientificamerican.com/article/photographic-memory-wearable/Google Scholar
Cross Ref
- Radu-Daniel Vatavu. 2012. Presence bubbles: supporting and enhancing human-human interaction with ambient media. Multimedia Tools and Applications, Vol. 58, 2 (May 2012), 371--383. Google Scholar
Digital Library
- Seung-Ho Yang, Hyun-Woo Kim, and Min Young Kim. 2011. Human Visual Augmentation Using Wearable Glasses with Multiple Cameras and Information Fusion of Human Eye Tracking and Scene Understanding. In Proceedings of the 6th International Conference on Human-robot Interaction (HRI '11). ACM, New York, NY, USA, 287--288. Google Scholar
Digital Library
- Hinbarji Z., Albatal R., O'Connor N., and Gurrin C. 2016. LoggerMan, a Comprehensive Logging and Visualization Tool to Capture Computer Usage. Vol. 9517. Springer, Cham.Google Scholar
- Yuhang Zhao, Sarit Szpiro, and Shiri Azenkot. 2015. ForeSee: A Customizable Head-Mounted Vision Enhancement System for People with Low Vision. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '15). ACM, New York, NY, USA, 239--249. Google Scholar
Digital Library
- Yuhang Zhao, Sarit Szpiro, Jonathan Knighten, and Shiri Azenkot. 2016. CueSee: Exploring Visual Cues for People with Low Vision to Facilitate a Visual Search Task. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 73--84.Google Scholar
Digital Library
- Liting Zhou, Zaher Hinbarji, Duc-Tien Dang-Nguyen, and Cathal Gurrin. 2018. LIFER: An Interactive Lifelog Retrieval System. In Proceedings of the 2018 ACM Workshop on The Lifelog Search Challenge (LSC '18). ACM, New York, NY, USA, 9--14.Google Scholar
Digital Library
- Floriano Zini, Martin Reinstadler, and Francesco Ricci. 2015. Life-logs Aggregation for Quality of Life Monitoring. In Proceedings of the 5th International Conference on Digital Health 2015 (DH '15). ACM, New York, NY, USA, 131--132. Google Scholar
Digital Library
Index Terms
Life-Tags: A Smartglasses-based System for Recording and Abstracting Life with Tag Clouds
Recommendations
Aggregating Life Tags for Opportunistic Crowdsensing with Mobile and Smartglasses Users
GoodTechs '20: Proceedings of the 6th EAI International Conference on Smart Objects and Technologies for Social GoodWe discuss in this work the opportunity of employing lifelogging devices, applications, and systems, such as systems that collect, process, and store video using mobile and wearable cameras, in order to run queries about objects and concepts of interest ...
The colour of life: novel visualisations of population lifestyles
MM '10: Proceedings of the 18th ACM international conference on MultimediaColour permeates our daily lives, yet we rarely take notice of it. In this work we utilise the SenseCam (a visual lifelogging tool), to investigate the predominant colours in one million minutes of human life that a group of 20 individuals encounter ...
Using visual lifelogs to automatically characterize everyday activities
Visual lifelogging is the term used to describe recording our everyday lives using wearable cameras, for applications which are personal to us and do not involve sharing our recorded data. Current applications of visual lifelogging are built around ...






Comments