skip to main content
research-article

Facilitating Human Activity Data Annotation via Context-Aware Change Detection on Smartwatches

Authors Info & Claims
Published:11 January 2021Publication History
Skip Abstract Section

Abstract

Annotating activities of daily living (ADL) is vital for developing machine learning models for activity recognition. In addition, it is critical for self-reporting purposes such as in assisted living where the users are asked to log their ADLs. However, data annotation becomes extremely challenging in real-world data collection scenarios, where the users have to provide annotations and labels on their own. Methods such as self-reports that rely on users’ memory and compliance are prone to human errors and become burdensome since they increase users’ cognitive load. In this article, we propose a light yet effective context-aware change point detection algorithm that is implemented and run on a smartwatch for facilitating data annotation for high-level ADLs. The proposed system detects the moments of transition from one to another activity and prompts the users to annotate their data. We leverage freely available Bluetooth low energy (BLE) information broadcasted by various devices to detect changes in environmental context. This contextual information is combined with a motion-based change point detection algorithm, which utilizes data from wearable motion sensors, to reduce the false positives and enhance the system's accuracy. Through real-world experiments, we show that the proposed system improves the quality and quantity of labels collected from users by reducing human errors while eliminating users’ cognitive load and facilitating the data annotation process.

References

  1. M. Sandberg, J. Kristensson, P. Midlöv, C. Fagerström, and U. Jakobsson. 2012. Prevalence and predictors of healthcare utilization among older people (60+): Focusing on ADL dependency and risk of depression. Archives of Gerontology and Geriatrics 54, 3 (2012), e349--e363.Google ScholarGoogle ScholarCross RefCross Ref
  2. D. Wang, J. Zheng, M. Kurosawa, Y. Inaba, and N. Kato. 2009. Changes in activities of daily living (ADL) among elderly Chinese by marital status, living arrangement, and availability of healthcare over a 3-year period. Environmental Health and Preventive Medicine 14, 2 (2009), 128.Google ScholarGoogle ScholarCross RefCross Ref
  3. P. Bharti, D. De, S. Chellappan, and S. K. Das. 2019. HuMAn: Complex activity recognition with multi-modal multi-positional body sensing. IEEE Transactions on Mobile Computing 18, 4 (2019), 857--870.Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. L. G. Clift, J. Lepley, H. Hagras, and A. F. Clark. 2018. Autonomous computational intelligence-based behaviour recognition in security and surveillance. In Proceedings of Counterterrorism, Crime Fighting, Forensics, and Surveillance Technologies II (2018), Vol. 10802. 108020L.Google ScholarGoogle ScholarCross RefCross Ref
  5. K. Davis et al. 2016. Activity recognition based on inertial sensors for ambient assisted living. In Proceedings of the 2016 19th International Conference on Information Fusion (Fusion). 371--378.Google ScholarGoogle Scholar
  6. A. Akbari, J. Wu, R. Grimsley, and R. Jafari. 2018. Hierarchical signal segmentation and classification for accurate activity recognition. In Proceedings of the 2018 ACM International Joint Conference and 2018 International Symposium on Pervasive and Ubiquitous Computing and Wearable Computers. 1596--1605.Google ScholarGoogle Scholar
  7. A. Akbari and R. Jafari. 2019. Transferring activity recognition models for new wearable sensors with deep generative domain adaptation. In Proceedings of the 18th International Conference on Information Processing in Sensor Networks. 85--96.Google ScholarGoogle Scholar
  8. F. Ordóñez and D. Roggen. 2016. Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16, 1 (2016), 115.Google ScholarGoogle ScholarCross RefCross Ref
  9. B. Hagsten, O. Svensson, and A. Gardulf. 2006. Health-related quality of life and self-reported ability concerning ADL and IADL after hip fracture: A randomized trial. Acta Orthopaedica 77, 1 (2006), 114--119.Google ScholarGoogle ScholarCross RefCross Ref
  10. S. Aminikhanghahi, T. Wang, and D. J. Cook. 2018. Real-time change point detection with application to smart home time series data. IEEE Transactions on Knowledge and Data Engineering 31 (2018), 1010--1023.Google ScholarGoogle ScholarCross RefCross Ref
  11. R. Solis, A. Pakbin, A. Akbari, B. J. Mortazavi, and R. Jafari. 2019. A human-centered wearable sensing platform with intelligent automated data annotation capabilities. In Proceedings of the International Conference on Internet of Things Design and Implementation. 255--260.Google ScholarGoogle Scholar
  12. A. Akbari and R. Jafari. 2020. Personalizing activity recognition models with quantifying different types of uncertainty using wearable sensors. In IEEE Transactions on Biomedical Engineering 67, 9 (2020), 2530--2541. DOI:10.1109/TBME.2019.2963816Google ScholarGoogle ScholarCross RefCross Ref
  13. S. Aminikhanghahi, M. Schmitter-Edgecombe, and D. J. Cook. 2019. Context-aware delivery of ecological momentary assessment. IEEE Journal of Biomedical and Health Informatics 24, 4 (2019), 1206--1214.Google ScholarGoogle ScholarCross RefCross Ref
  14. T. Patterson et al. 2016. Sensor-based change detection for timely solicitation of user engagement. IEEE Transactions on Mobile Computing 16, 10 (2016), 2889--2900.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Y. Kawahara and M. Sugiyama. 2012. Sequential change-point detection based on direct density-ratio estimation. Statistical Analysis and Data Mining: The ASA Data Science Journal 5, 2 (2012), 114--127.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. T. Patterson, S. McClean, C. Nugent, S. Zhang, L. Galway, and I. Cleland. 2014. Online change detection for timely solicitation of user interaction. In Proceedings of the International Conference on Ubiquitous Computing and Ambient Intelligence. 116--123.Google ScholarGoogle Scholar
  17. T. R. Bennett, H. C. Massey, J. Wu, S. A. Hasnain, and R. Jafari. 2016. MotionSynthesis toolset (MoST): An open source tool and data set for human motion data synthesis and validation. IEEE Sensors Journal 16, 13 (2016), 5365--5375.Google ScholarGoogle ScholarCross RefCross Ref
  18. D. Roggen et al. 2010. Collecting complex activity datasets in highly rich networked sensor environments. In Proceedings of the 2010 7th International Conference on Networked Sensing Systems (INSS’2010). 233--240.Google ScholarGoogle ScholarCross RefCross Ref
  19. H. Gjoreski et al. 2018. The university of Sussex-Huawei locomotion and transportation dataset for multimodal analytics with mobile devices. IEEE Access 6 (2018), 42592--42604.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. T. Diethe, N. Twomey, and P. A. Flach. 2016. Active transfer learning for activity recognition. In Proceedings of the European Symposium on Artificial Neural Networks (ESANN’16).Google ScholarGoogle Scholar
  21. S. Sabato and T. Hess. 2016. Interactive algorithms: From pool to stream. In Proceedings of the Conference on Learning Theory (2016). 1419--1439.Google ScholarGoogle Scholar
  22. A. Akbari, R. Solis Castilla, R. Jafari, and B. J. Mortazavi. 2020. Using intelligent personal annotations to improve human activity recognition for movements in natural environments. IEEE Journal of Biomedical and Health Informatics (JBHI) 24, 9 (2020), 2639--2650. DOI:10.1109/JBHI.2020.2966151Google ScholarGoogle ScholarCross RefCross Ref
  23. T. Sztyler and H. Stuckenschmidt. 2017. Online personalization of cross-subjects based activity recognition models on wearable devices. In Proceedings of the 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom’17). 180--189.Google ScholarGoogle Scholar
  24. I. Cleland et al. 2014. Evaluation of prompted annotation of activity data recorded from a smart phone. Sensors 14, 9 (2014), 15861--15879.Google ScholarGoogle ScholarCross RefCross Ref
  25. S. Reddy, M. Mun, J. Burke, D. Estrin, M. Hansen, and M. Srivastava. 2010. Using mobile phones to determine transportation modes. ACM Transactions on Sensor Networks (TOSN) 6, 2 (2010).Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. J. Muñoz-Marí, F. Bovolo, L. Gómez-Chova, L. Bruzzone, and G. Camp-Valls. 2010. Semisupervised one-class support vector machines for classification of remote sensing data. IEEE Transactions on Geoscience and Remote Sensing 48, 8 (2010), 3188--3197.Google ScholarGoogle ScholarCross RefCross Ref
  27. S. Aminikhanghahi and D. J. Cook. 2017. A survey of methods for time series change point detection. Knowledge and Information Systems 51, 2 (2017), 339--367.Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. D. R. Jeske, V. M. De Oca, W. Bischoff, and M. Marvasti. 2009. Cusum techniques for timeslot sequences with applications to network surveillance. Computational Statistics 8 Data Analysis 53, 12 (2009), 4332--4344.Google ScholarGoogle Scholar
  29. X. Song, M. Wu, C. Jermaine, and S. Ranka. 2007. Statistical change detection for multi-dimensional data. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2007). 667--676.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. L. I. Kuncheva and W. J. Faithfull. 2013. PCA feature extraction for change detection in multidimensional unlabeled data. IEEE Transactions on Neural Networks and Learning Systems 25, 1 (2013), 69--80.Google ScholarGoogle ScholarCross RefCross Ref
  31. Y. Kawahara and M. Sugiyama. 2009. Change-point detection in time-series data by direct density-ratio estimation. In Proceedings of the 2009 SIAM International Conference on Data Mining (2009). 389--400.Google ScholarGoogle Scholar
  32. S. Liu, M. Yamada, N. Collier, and M. Sugiyama. 2013. Change-point detection in time-series data by relative density-ratio estimation. Neural Networks 43 (2013), 72--83.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. A. A. Qahtan, B. Alharbi, S. Wang, and X. Zhang. 2015. A PCA-based change detection framework for multidimensional data streams: Change detection in multidimensional data streams. In Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015). 935--944.Google ScholarGoogle Scholar
  34. J. Wu and R. Jafari. 2018. Orientation independent activity/gesture recognition using wearable motion sensors. IEEE Internet of Things Journal 6, 2 (2018), 1427--1437.Google ScholarGoogle ScholarCross RefCross Ref
  35. E. R. Sykes, S. Pentland, and S. Nardi. 2015. Context-aware mobile apps using iBeacons: Towards smarter interactions. In Proceedings of the 25th Annual International Conference on Computer Science and Software Engineering (2015). 120--129.Google ScholarGoogle Scholar
  36. M. Levandowsky and D. Winter. 1971. Distance between sets. Nature 234, 5323 (1971), 34--35.Google ScholarGoogle Scholar
  37. Polar M600 | Sport smart watch for fitness | Polar Global. Retrieved on March 2019 from https://www.polar.com/blog/polar-m600-android-wear-2-0-sports-smartwatch/.Google ScholarGoogle Scholar

Index Terms

  1. Facilitating Human Activity Data Annotation via Context-Aware Change Detection on Smartwatches

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!