Scalable edge-based hyperdimensional learning system with brain-like neural adaptation

ABSTRACT

In the Internet of Things (IoT) domain, many applications are running machine learning algorithms to assimilate the data collected in the swarm of devices. Sending all data to the powerful computing environment, e.g., cloud, poses significant efficiency and scalability issues. A promising way is to distribute the learning tasks onto the IoT hierarchy, often referred to edge computing; however, the existing sophisticated algorithms such as deep learning are often overcomplex to run on less-powerful and unreliable embedded IoT devices. Hyperdimensional Computing (HDC) is a brain-inspired learning approach for efficient and robust learning on today's embedded devices. Encoding, or transforming the input data into high-dimensional representation, is the key first step of HDC before performing a learning task. All existing HDC approaches use a static encoder; thus, they still require very high dimensionality, resulting in significant efficiency loss for the edge devices with limited resources. In this paper, we have developed NeuralHD, a new HDC approach with a dynamic encoder for adaptive learning. Inspired by human neural regeneration study in neuroscience, NeuralHD identifies insignificant dimensions and regenerates those dimensions to enhance the learning capability and robustness. We also present a scalable learning framework to distribute NeuralHD computation over edge devices in IoT systems. Our solution enables edge devices capable of real-time learning from both labeled and unlabeled data. Our evaluation on a wide range of practical classification tasks shows that NeuralHD provides 5.7X and 6.1X (12.3X and 14.1X) faster and more energy-efficient training compared to the HD-based algorithms (DNNs) running on the same platform. NeuralHD also provides 4.2X and 11.6X higher robustness to noise in the unreliable network and hardware of IoT environments as compared to DNNs.

References

1. G. Zhu, D. Liu, Y. Du, C. You, J. Zhang, and K. Huang, "Toward an intelligent edge: wireless communication meets machine learning," IEEE Communications Magazine, vol. 58, no. 1, pp. 19--25, 2020.
2. X. Wang, Y. Han, C. Wang, Q. Zhao, X. Chen, and M. Chen, "In-edge ai: Intelligentizing mobile edge computing, caching and communication by federated learning," arXiv preprint arXiv:1809.07857, 2018.
3. V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, "Federated multi-task learning," in Advances in Neural Information Processing Systems, pp. 4424--4434, 2017.
4. E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, "How to backdoor federated learning," arXiv preprint arXiv:1807.00459, 2018.
5. Y. Kang, J. Hauswald, C. Gao, A. Rovinski, T. Mudge, J. Mars, and L. Tang, "Neurosurgeon: Collaborative intelligence between the cloud and mobile edge," ACM SIGPLAN Notices, vol. 52, no. 4, pp. 615--629, 2017.
6. J. Pan and J. McElhannon, "Future edge cloud and edge computing for internet of things applications," IEEE Internet of Things Journal, vol. 5, no. 1, pp. 439--449, 2017.
7. H. Li, K. Ota, and M. Dong, "Learning iot in edge: Deep learning for the internet of things with edge computing," IEEE network, vol. 32, no. 1, pp. 96--101, 2018.
8. O. Yilmaz, "Symbolic computation using cellular automata-based hyperdimensional computing," Neural computation, vol. 27, no. 12, pp. 2661--2692, 2015.
9. P. Kanerva, "Hyperdimensional computing: An introduction to computing in distributed representation with high-dimensional random vectors," Cognitive Computation, vol. 1, no. 2, pp. 139--159, 2009.
10. P. Kanerva, J. Kristofersson, and A. Holst, "Random indexing of text samples for latent semantic analysis," in Proceedings of the 22nd annual conference of the cognitive science society, vol. 1036, Citeseer, 2000.
11. P. Poduval et al., "Stochd: Stochastic hyperdimensional system for efficient and robust learning from raw data," in IEEE/ACM Design Automation Conference (DAC), 2021.
12. M. Imani, A. Rahimi, D. Kong, T. Rosing, and J. M. Rabaey, "Exploring hyper-dimensional associative memory," in High Performance Computer Architecture (HPCA), 2017 IEEE International Symposium on, pp. 445--456, IEEE, 2017.
13. T. F. Wu, H. Li, P.-C. Huang, A. Rahimi, J. M. Rabaey, H.-S. P. Wong, M. M. Shulaker, and S. Mitra, "Brain-inspired computing exploiting carbon nanotube fets and resistive ram: Hyperdimensional computing case study," in 2018 IEEE International Solid-State Circuits Conference-(ISSCC), pp. 492--494, IEEE, 2018.
14. A. Mitrokhin, P. Sutor, C. Fermuller, and Y. Aloimonos, "Learning sensorimotor control with neuromorphic sensors: Toward hyperdimensional active perception," Science Robotics, vol. 4, no. 30, p. eaaw6736, 2019.
15. A. Hérnandez-Cano et al., "Reghd: Robust and efficient regression in hyper-dimensional learning system," in IEEE/ACM Design Automation Conference (DAC), 2021.
16. P. Poduval et al., "Cognitive correlative encoding for genome sequence matching in hyperdimensional system," in IEEE/ACM Design Automation Conference (DAC), 2021.
17. A. Mitrokhin, P. Sutor, D. Summers-Stay, C. Fermüller, and Y. Aloimonos, "Symbolic representation and learning with hyperdimensional computing,"
18. O. Räsänen and S. Kakouros, "Modeling dependencies in multiple parallel data streams with hyperdimensional computing," IEEE Signal Processing Letters, vol. 21, no. 7, pp. 899--903, 2014.
19. O. Rasanen and J. Saarinen, "Sequence prediction with sparse distributed hyperdimensional coding applied to the analysis of mobile phone use patterns," IEEE Transactions on Neural Networks and Learning Systems, vol. PP, no. 99, pp. 1--12, 2015.
20. A. Joshi, J. Halseth, and P. Kanerva, "Language geometry using random indexing," Quantum Interaction 2016 Conference Proceedings, In press.
21. S. Jockel, "Crossmodal learning and prediction of autobiographical episodic experiences using a sparse distributed memory," 2010.
22. L. Ge and K. K. Parhi, "Classification using hyperdimensional computing: A review," arXiv preprint arXiv:2004.11204, 2020.
23. H. Li, T. F. Wu, A. Rahimi, K.-S. Li, M. Rusch, C.-H. Lin, J.-L. Hsu, M. M. Sabry, S. B. Eryilmaz, J. Sohn, et al., "Hyperdimensional computing with 3d vrram in-memory kernels: Device-architecture co-design for energy-efficient, error-resilient language recognition," in Electron Devices Meeting (IEDM), 2016 IEEE International, pp. 16--1, IEEE, 2016.
24. P. Poduval et al., "Hyperdimensional self-learning systems robust to technology noise and bit-flip attacks," in IEEE/ACM International Conference on Computer-Aided Design (ICCAD), IEEE, 2021.
25. M. Imani, Y. Kim, S. Riazi, J. Messerly, P. Liu, F. Koushanfar, and T. Rosing, "A framework for collaborative learning in secure high-dimensional space," in 2019 IEEE 12th International Conference on Cloud Computing (CLOUD), pp. 435--446, IEEE, 2019.
26. A. Hérnandez-Cano et al., "Prid: Model inversion privacy attacks in hyperdimensional learning systems," in IEEE/ACM Design Automation Conference (DAC), 2021.
27. A. Rahimi, P. Kanerva, and J. M. Rabaey, "A robust and energy-efficient classifier using brain-inspired hyperdimensional computing," in Proceedings of the 2016 International Symposium on Low Power Electronics and Design, pp. 64--69, 2016.
28. M. Imani, Z. Zou, S. Bosch, S. A. Rao, S. Salamat, V. Kumar, Y. Kim, and T. Rosing, "Revisiting hyperdimensional learning for fpga and low-power architectures," in 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA), pp. 221--234, IEEE, 2021.
29. M. Imani, J. Morris, J. Messerly, H. Shu, Y. Deng, and T. Rosing, "Bric: Locality-based encoding for energy-efficient brain-inspired hyperdimensional computing," in Proceedings of the 56th Annual Design Automation Conference 2019, pp. 1--6, 2019.
30. J. H. Morrison and P. R. Hof, "Life and death of neurons in the aging brain," Science, vol. 278, no. 5337, pp. 412--419, 1997.
31. E. I. Rugarli and T. Langer, "Mitochondrial quality control: a matter of life and death for neurons," The EMBO journal, vol. 31, no. 6, pp. 1336--1349, 2012.
32. F. H. Gage and S. Temple, "Neural stem cells: generating and regenerating the brain," Neuron, vol. 80, no. 3, pp. 588--601, 2013.
33. L. Gao, W. Guan, M. Wang, H. Wang, J. Yu, Q. Liu, B. Qiu, Y. Yu, Y. Ping, X. Bian, et al., "Direct generation of human neuronal cells from adult astrocytes by small molecules," Stem cell reports, vol. 8, no. 3, pp. 538--547, 2017.
34. G. Stoll and H. W. Müller, "Nerve injury, axonal degeneration and neural regeneration: basic insights," Brain pathology, vol. 9, no. 2, pp. 313--325, 1999.
35. "Number of new generated neurons every day, Nicolas Toni." https://wp.unil.ch/discoverunil/2017/06/we-create-1500-new-neurons-every-day/.
36. S. Ackerman et al., Discovering the brain. National Academies Press, 1992.
37. Z. Zou et al., "Manihd: Efficient hyper-dimensional learning using manifold trainable encoder," in 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 850--855, IEEE, 2021.
38. M. Imani, S. Bosch, M. Javaheripi, B. Rouhani, X. Wu, F. Koushanfar, and T. Rosing, "Semihd: Semi-supervised learning using hyperdimensional computing," in 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1--8, IEEE, 2019.
39. P. Kanerva, "Encoding structure in boolean space," in ICANN 98, pp. 387--392, Springer, 1998.
40. B. Pakkenberg, D. Pelvig, L. Marner, M. J. Bundgaard, H. J. G. Gundersen, J. R. Nyengaard, and L. Regeur, "Aging and the human neocortex," Experimental gerontology, vol. 38, no. 1--2, pp. 95--99, 2003.
41. B. B. Andersen, H. J. G. Gundersen, and B. Pakkenberg, "Aging of the human cerebellum: a stereological study," Journal of Comparative Neurology, vol. 466, no. 3, pp. 356--365, 2003.
42. A. Rahimi and B. Recht, "Random features for large-scale kernel machines" in Advances in neural information processing systems, pp. 1177--1184, 2008.
43. B. Scholkopf, K.-K. Sung, C.J. Burges, F. Girosi, P. Niyogi, T. Poggio, and V. Vapnik, "Comparing support vector machines with gaussian kernels to radial basis function classifiers," IEEE transactions on Signal Processing, vol. 45, no. 11, pp. 2758--2765, 1997.
44. L. Marner, J. R. Nyengaard, Y. Tang, and B. Pakkenberg, "Marked loss of myelinated nerve fibers in the human brain with age," Journal of comparative neurology, vol. 462, no. 2, pp. 144--152, 2003.
45. M. F. Paredes, S. F. Sorrells, A. Cebrian-Silla, K. Sandoval, D. Qi, K. W. Kelley, D. James, S. Mayer, J. Chang, K. I. Auguste, et al., "Does adult neurogenesis persist in the human hippocampus?," Cell Stem Cell, vol. 23, no. 6, pp. 780--781, 2018.
46. K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, "Practical secure aggregation for privacy-preserving machine learning," in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, pp. 1175--1191, 2017.
47. T. Feist, "Vivado design suite," White Paper, vol. 5, 2012.
48. S. Salamat, M. Imani, B. Khaleghi, and T. Rosing, "F5-hd: Fast flexible fpga-based framework for refreshing hyperdimensional computing," in FPGA, pp. 53--62, ACM, 2019.
49. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278--2324, 1998.
50. D. Ciregan, U. Meier, and J. Schmidhuber, "Multi-column deep neural networks for image classification," in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pp. 3642--3649, IEEE, 2012.
51. "Uci machine learning repository." http://archive.ics.uci.edu/ml/datasets/ISOLET.
52. D. Anguita, A. Ghio, L. Oneto, X. Parra, and J. L. Reyes-Ortiz, "Human activity recognition on smartphones using a multiclass hardware-friendly support vector machine," in International workshop on ambient assisted living, pp. 216--223, Springer, 2012.
53. A. Angelova, Y. Abu-Mostafam, and P. Perona, "Pruning training sets for learning of object categories," in Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on, IEEE, 2005.
54. "Pecan street dataport." https://dataport.cloud/.
55. A. Reiss and D. Stricker, "Introducing a new benchmarked dataset for activity monitoring," in Wearable Computers (ISWC), 2012 16th International Symposium on, pp. 108--109, IEEE, 2012.
56. M. Zaharia, R. S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen, S. Venkataraman, M. J. Franklin, et al., "Apache spark: a unified engine for big data processing," Communications of the ACM, vol. 59, no. 11, pp. 56--65, 2016.
57. Y. Kim, P. Mercati, A. More, E. Shriver, and T. Rosing, "P4: Phase-based power/performance prediction of heterogeneous systems via neural networks," in Computer-Aided Design (ICCAD), 2017 IEEE/ACM International Conference on, pp. 683--690, IEEE, 2017.
58. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al., "Tensorflow: Large-scale machine learning on heterogeneous distributed systems," arXiv preprint arXiv:1603.04467, 2016.
59. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, "Optuna: A next-generation hyperparameter optimization framework," in Proceedings of the 25th ACMSIGKDD international conference on knowledge discovery & data mining, pp. 2623--2631, 2019.
60. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al., "Scikit-learn: Machine learning in python," Journal of Machine Learning Research, vol. 12, no. Oct, pp. 2825--2830, 2011.
61. M. Imani, J. Messerly, F. Wu, W. Pi, and T. Rosing, "A binary learning framework for hyperdimensional computing," in 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 126--131, IEEE, 2019.
62. H. Sharma, J. Park, D. Mahajan, E. Amaro, J. K. Kim, C. Shao, A. Mishra, and H. Esmaeilzadeh, "From high-level deep neural models to fpgas," in Microarchitecture (MICRO), 2016 49th Annual IEEE/ACM International Symposium on, pp. 1--12, IEEE, 2016.
63. T. Geng, T. Wang, A. Sanaullah, C. Yang, R. Xu, R. Patel, and M. Herbordt, "Fpdeep: Acceleration and load balancing of cnn training on fpga clusters," in 2018 IEEE 26th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pp. 81--84, IEEE, 2018.
64. S.-H. Lee, "Technology scaling challenges and opportunities of memory devices," in 2016 IEEE International Electron Devices Meeting (IEDM), pp. 1--1, IEEE, 2016.
65. K. T. Lee, W. Kang, E.-A. Chung, G. Kim, H. Shim, H. Lee, H. Kim, M. Choe, N.-I. Lee, A. Patel, et al., "Technology scaling on high-k & metal-gate finfet bti reliability," in 2013 IEEE International Reliability Physics Symposium (IRPS), pp. 2D-1, IEEE, 2013.
66. H. Esmaeilzadeh, E. Blem, R. S. Amant, K. Sankaralingam, and D. Burger, "Dark silicon and the end of multicore scaling," in Computer Architecture (ISCA), 2011 38th Annual International Symposium on, pp. 365--376, IEEE, 2011.
67. S. Yi, Z. Hao, Z. Qin, and Q. Li, "Fog computing: Platform and applications," in 2015 Third IEEE Workshop on Hot Topics in Web Systems and Technologies (HotWeb), pp. 73--78, IEEE, 2015.
68. L. Tong, Y. Li, and W. Gao, "A hierarchical edge cloud architecture for mobile computing," in INFOCOM 2016-The 35th Annual IEEE International Conference on Computer Communications, IEEE, pp. 1--9, IEEE, 2016.
69. P. Garcia Lopez, A. Montresor, D. Epema, A. Datta, T. Higashino, A. Iamnitchi, M. Barcellos, P. Felber, and E. Riviere, "Edge-centric computing: Vision and challenges," ACM SIGCOMM Computer Communication Review, vol. 45, no. 5, pp. 37--42, 2015.
70. J. H. Ko, T. Na, M. F. Amir, and S. Mukhopadhyay, "Edge-host partitioning of deep neural networks with feature space encoding for resource-constrained internet-of-things platforms," arXiv preprint arXiv:1802.03835, 2018.
71. J. Venkatesh, C. Chan, A. S. Akyurek, and T. S. Rosing, "A modular approach to context-aware iot applications," in Internet-of-Things Design and Implementation (IoTDI), 2016 IEEE First International Conference on, pp. 235--240, IEEE, 2016.
72. H. Grunert and A. Heuer, "Rewriting complex queries from cloud to fog under capability constraints to protect the users' privacy," Open Journal of Internet Of Things (OJIOT), vol. 3, no. 1, pp. 31--45, 2017.
73. "Aws greengrass." https://aws.amazon.com/greengrass/.
74. B. Khaleghi, A. Khamis, F. O. Karray, and S. N. Razavi, "Multisensor data fusion: A review of the state-of-the-art," Information fusion, vol. 14, no. 1, pp. 28--44, 2013.
75. S. Qaisar, R. M. Bilal, W. Iqbal, M. Naureen, and S. Lee, "Compressive sensing: From theory to applications, a survey," Journal of Communications and networks, vol. 15, no. 5, pp. 443--456, 2013.
76. S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
77. P. Kanerva, "What we mean when we say "what's the dollar of mexico?": Prototypes and mapping in concept space," in AAAI Fall Symposium: Quantum Informatics for Cognitive, Social, and Semantic Processes, pp. 2--6, 2010.
78. P. Kanerva, J. Kristofersson, and A. Holst, "Random indexing of text samples for latent semantic analysis," in Proceedings of the 22nd annual conference of the cognitive science society, vol. 1036, Citeseer, 2000.
79. A. Hernández-Cano et al., "A framework for efficient and binary clustering in high-dimensional space," in 2021 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1859--1864, IEEE, 2021.
80. A. Burrello, K. Schindler, L. Benini, and A. Rahimi, "One-shot learning for ieeg seizure detection using end-to-end binary operations: Local binary patterns with hyperdimensional computing," in 2018 IEEE Biomedical Circuits and Systems Conference (BioCAS), pp. 1--4, IEEE, 2018.
81. D. Kleyko, A. Rahimi, D. A. Rachkovskij, E. Osipov, and J. M. Rabaey, "Classification and recall with binary hyperdimensional computing: Tradeoffs in choice of density and mapping characteristics," IEEE transactions on neural networks and learning systems, no. 99, pp. 1--19, 2018.
82. M. Imani, D. Kong, A. Rahimi, and T. Rosing, "Voicehd: Hyperdimensional computing for efficient speech recognition," in 2017 IEEE international conference on rebooting computing (ICRC), pp. 1--8, IEEE, 2017.
83. M. Imani, S. Bosch, S. Datta, S. Ramakrishna, S. Salamat, J. M. Rabaey, and T. Rosing, "Quanthd: A quantization framework for hyperdimensional computing," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2019.
84. T. Wu, P. Huang, A. Rahimi, H. Li, J. Rabaey, P. Wong, and S. Mitra, "Brain-inspired computing exploiting carbon nanotube fets and resistive ram: Hyperdimensional computing case study," in IEEE Intl. Solid-State Circuits Conference (ISSCC), IEEE, 2018.
85. M. Imani, S. Pampana, S. Gupta, M. Zhou, Y. Kim, and T. Rosing, "Dual: Acceleration of clustering algorithms using digital-based processing in-memory," in 2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pp. 356--371, IEEE, 2020.
86. A. Kazemi et al., "Mimhd: Accurate and efficient hyperdimensional inference using multi-bit in-memory computing," in 2021 IEEE/ACM International Symposium on Low Power Electronics and Design (ISLPED), pp. 1--6, IEEE, 2021.
87. M. Imani, X. Yin, J. Messerly, S. Gupta, M. Niemier, X. S. Hu, and T. Rosing, "Searchd: A memory-centric hyperdimensional computing with stochastic training," IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 39, no. 10, pp. 2422--2433, 2019.

Index Terms

• Computer systems organization

  • Architectures

    • Other architectures

• Computing methodologies

  • Machine learning

    • Learning paradigms

    • Machine learning approaches

      • Neural networks

• Software and its engineering