Inferring finer-grained human information with multi-modal cross-granularity learning: PhD forum abstract

ABSTRACT

Existing machine learning algorithms for human information inference are typically data-driven models trained on carefully labeled datasets. Given the significant labeling effort, traditional pure data-driven approaches are challenging to implement for emerging smart applications requiring long-term finer-grained information. Taking activities of daily life (ADL) tracking for elders as an example, prior work mostly focused on context-level information learning such as cooking and cleaning. [8]. However, new applications such as evaluating elders' cognitive impairments progress by tracking their ADL engagement requires finer-grained, i.e., action-level information [7]. In practice, labeling the day-length data at such granularity can be very expensive and requires a lot of human efforts [9]. My research focuses on the inference problems in the scope of human physical condition monitoring and activity recognition with limited labeled data. To alleviate the effort of labeling large amounts of data, prior works on semi-supervised learning combine a small amount of labeled data with a large amount of unlabeled data to train the model. However, as the label granularity (number of classes) increasing, the difficulty to distinguish nuance distinctions between finer-grained classes escalates as well. This makes training a robust semi-supervised model for finer-grained classification with less labels difficult if not impossible. Fortunately, coarse-grained (context-level) labels is usually available or cheaper to obtain in practice. In this case, the multi-granularity hierarchy between finer and coarse labels follows the aggregation relation defined in [5]. This hierarchical relation can be leveraged in the tasks of inferring finer-grained information. In addition, it is illustrated by the previous study that co-located multi modality sensing systems capture complementary aspects of the same event [6]. The research question I focus on is how to infer finer-grained human information with coarse-grained labeled data leveraging complementary multi-modal sensing? I target three directions: 1) a cross-granularity semi-supervised setting: how to utilize coarse-grained labeled data with a small amount of finer-grained labeled data to infer finer-grained human information, 2) cross-granularity relationship learning: how to learn the multi-granularity class hierarchy from data and further help the finer-grained human information acquisition, 3) enhancing inference granularity by leveraging multi-model sensing: how to leverage the complimentary co-located multiple sensing modalities to accurately infer finer-grained human information?

References

1. Carlos Avilés-Cruz, Andrés Ferreyra-Ramírez, Arturo Zúñiga-López, and Juan Villegas-Cortéz. 2019. Coarse-fine convolutional deep-learning strategy for human activity recognition. Sensors 19, 7 (2019), 1556.
2. Pratool Bharti, Debraj De, Sriram Chellappan, and Sajal K Das. 2018. HuMAn: Complex activity recognition with multi-modal multi-positional body sensing. IEEE Transactions on Mobile Computing 18, 4 (2018), 857--870.
3. Jun He, Qian Zhang, Liqun Wang, and Ling Pei. 2018. Weakly supervised human activity recognition from wearable sensors by recurrent attention learning. IEEE Sensors Journal 19, 6 (2018), 2287--2297.
4. Zhizhang Hu, Tong Yu, Yue Zhang, and Shijia Pan. 2020. Fine-grained activities recognition with coarse-grained labeled multi-modal data. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers. 644--649.
5. Gordon McCalla, Jim Greer, Bryce Barrie, and Paul Pospisil. 1992. Granularity hierarchies. Computers & Mathematics with Applications 23, 2--5 (1992), 363--375.
6. Shijia Pan, Mario Berges, Juleen Rodakowski, Pei Zhang, and Hae Young Noh. 2019. Fine-grained recognition of activities of daily living through structural vibration and electrical sensing. In Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation. 149--158.
7. Juleen Rodakowski, Elizabeth R Skidmore, Charles F Reynolds III, Mary Amanda Dew, Meryl A Butters, Margo B Holm, Oscar L Lopez, and Joan C Rogers. 2014. Can performance on daily activities discriminate between older adults with normal cognitive function and those with mild cognitive impairment? Journal of the American Geriatrics Society 62, 7 (2014), 1347--1352.
8. Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian Förster, Gerhard Tröster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, et al. 2010. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 Seventh international conference on networked sensing systems (INSS). IEEE, 233--240.
9. Maja Stikic, Diane Larlus, Sandra Ebert, and Bernt Schiele. 2011. Weakly supervised recognition of daily life activities with wearable sensors. IEEE transactions on pattern analysis and machine intelligence 33, 12 (2011), 2521--2537.
10. Otilia Stretcu, Emmanouil Antonios Platanios, Tom Mitchell, and Barnabás Póczos. 2020. Coarse-to-Fine Curriculum Learning for Classification. In International Conference on Learning Representations (ICLR) Workshop on Bridging AI and Cognitive Science (BAICS).
11. Ming Zeng, Tong Yu, Xiao Wang, Le T Nguyen, Ole J Mengshoel, and Ian Lane. 2017. Semi-supervised convolutional neural networks for human activity recognition. In 2017 IEEE International Conference on Big Data (Big Data). IEEE, 522--529.

Index Terms

• Human-centered computing

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile computing theory, concepts and paradigms

      • Ambient intelligence