Yu Zhang
ABSTRACT
Mobile Deep Learning (MDL) has emerged as a privacy-preserving learning paradigm for mobile devices. This paradigm offers unique features such as privacy preservation, continual learning and low-latency inference to the building of personal mobile sensing applications. However, squeezing Deep Learning to mobile devices is extremely challenging due to resource constraint. Traditional Deep Neural Networks (DNNs) are usually over-parametered, hence incurring huge resource overhead for on-device learning. In this paper, we present a novel on-device deep learning framework named MDLdroidLite that transforms traditional DNNs into resource-efficient model structures for on-device learning. To minimize resource overhead, we propose a novel Release-and-Inhibit Control (RIC) approach based on Model Predictive Control theory to efficiently grow DNNs from tiny to backbone. We also design a gate-based fast adaptation mechanism for channel-level knowledge transformation to quickly adapt new-born neurons with existing neurons, enabling safe parameter adaptation and fast convergence for on-device training. Our evaluations show that MDLdroidLite boosts on-device training on various PMS datasets with 28x to 50x less model parameters, 4x to 10x less floating number operations than the state-of-the-art model structures while keeping the same accuracy level.
- Davide A., Alessandro G., Luca O., Xavier P., and J L Reyes-Ortiz. 2013. A Public Domain Dataset for Human Activity Recognition using Smartphones. In ESANN'13.Google Scholar
- Tomás Angles, Raffaello Camoriano, Alessandro Rudi, and Lorenzo Rosasco. 2016. NYTRO: When Subsampling Meets Early Stopping.Google Scholar
- Oresti B., Rafael G., Juan A. H., Miguel D., Hector P., Ignacio R., Alejandro S., and Claudia V. 2014. mHealthDroid: A Novel Framework for Agile Development of Mobile Health Applications. In Ambient Assisted Living and Daily Activities.Google Scholar
- Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoc Le. 2018. Understanding and Simplifying One-Shot Architecture Search. In ICML '18.Google Scholar
- Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. 2017. Efficient Architecture Search by Network Transformation. In AAAI '17.Google Scholar
- T. Chai and R. R. Draxler. 2014. Root mean square error (RMSE) or mean absolute error (MAE)? - Arguments against avoiding RMSE in the literature. Geoscientific Model Development (2014).Google Scholar
- Kaixuan Chen, Dalin Zhang, Lina Yao, Bin Guo, Zhiwen Yu, and Yunhao Liu. 2020. Deep Learning for Sensor-based Human Activity Recognition: Overview, Challenges and Opportunities. (2020).Google Scholar
- Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. 2015. Net2Net: Accelerating Learning via Knowledge Transfer. (2015).Google Scholar
- Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. 2019. A Closer Look at Few-shot Classification. In ICLR'19.Google Scholar
- Yitao Chen, Saman Biookaghazadeh, and Ming Zhao. 2018. Exploring the Capabilities of Mobile Devices Supporting Deep Learning. In HPDC '18.Google Scholar
- Yanjiao Chen, Baolin Zheng, Zihan Zhang, Qian Wang, Chao Shen, and Qian Zhang. 2020. Deep Learning on Mobile and Embedded Devices: State-of-the-Art, Challenges, and Future Directions. ACM Comput. Surv. (2020).Google Scholar
- Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. 2017. A Survey of Model Compression and Acceleration for Deep Neural Networks. arXiv (2017).Google Scholar
- Luo Chunjie, Zhan jianfeng, Wang lei, and Yang Qiang. 2017. Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks. (2017).Google Scholar
- X. Dai, H. Yin, and N. K. Jha. 2019. NeST: ANeural Network Synthesis Tool Based on a Grow-and-Prune Paradigm. TC (2019).Google Scholar
- Lokenath Debnath and Kanadpriya Basu. 2015. A short history of probability theory and its applications. International Journal of Mathematical Education in Science and Technology (2015).Google ScholarCross Ref
- Yunbin Deng. 2019. Deep Learning on Mobile Devices - A Review. CoRR (2019).Google Scholar
- Sauptik Dhar, Junyao Guo, Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah. 2020. On-Device Machine Learning: An Algorithms and Learning Theory Perspective. (2020).Google Scholar
- Xiaocong Du, Zheng Li, and Yu Cao. 2019. CGaP: Continuous Growth and Pruning for Efficient Deep Learning. CoRR (2019).Google Scholar
- Jiemin Fang, Yuzhu Sun, Kangjian Peng, Qian Zhang, Yuan Li, Wenyu Liu, and Xinggang Wang. 2020. Fast Neural Network Adaptation via Parameter Remapping and Architecture Search. In ICLR '20.Google Scholar
- Jonathan Frankle and Michael Carbin. 2019. The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. In ICLR '19.Google Scholar
- Xavier Glorot and Y. Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. Journal of Machine Learning Research - Proceedings Track (2010).Google Scholar
- Taesik Gong, Yeonsu Kim, Jinwoo Shin, and Sung-Ju Lee. 2019. MetaSense: Few-Shot Adaptation to Untrained Conditions in Deep Mobile Sensing. In SenSys '19.Google ScholarDigital Library
- Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. 2020. Single Path One-Shot Neural Architecture Search with Uniform Sampling. (2020).Google Scholar
- Seungyeop Han, Haichen Shen, Matthai Philipose, Sharad Agarwal, Alec Wolman, and Arvind Krishnamurthy. 2016. MCDNN: An Approximation-Based Execution Framework for Deep Stream Processing Under Resource Constraints. In MobiSys '16.Google ScholarDigital Library
- Andrew Hard, Kanishka Rao, Rajiv Mathews, Françoise Beaufays, Sean Augenstein, Hubert Eichner, Chloé Kiddon, and Daniel Ramage. 2018. Federated Learning for Mobile Keyboard Prediction. CoRR (2018).Google Scholar
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In ICCV '15.Google ScholarDigital Library
- Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR (2017).Google Scholar
- Sergey Ioffe and Christian Szegedy. 2015. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In ICML '15.Google ScholarDigital Library
- Ozan Irsoy and Ethem Alpaydin. 2018. Continuously Constructive Deep Neural Networks. IEEE Transactions on Neural Networks and Learning Systems (2018).Google Scholar
- Jakub Konecný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik. 2016. Federated Optimization: Distributed Machine Learning for On-Device Intelligence. CoRR (2016).Google Scholar
- Alex Krizhevsky. 2009. Learning Multiple Layers of Features from Tiny Images.Google Scholar
- J. Leon Kröger, Philip R., and T. Rahman B. 2019. Privacy Implications of Accelerometer Data: A Review of Possible Inferences. In ICCSP '19.Google ScholarDigital Library
- Sergey L., Nadia K., Innokentiy K., Victor K., and Valeri A. M. 2018. Latent Factors Limiting the Performance of sEMG-Interfaces. Sensors (2018).Google Scholar
- N. D. Lane, S. Bhattacharya, A. Mathur, P. Georgiev, C. Forlivesi, and F. Kawsar. 2017. Squeezing Deep Learning into Mobile and Embedded Devices. IEEE Pervasive Computing (2017).Google Scholar
- Nicholas D. Lane and Petko Georgiev. 2015. Can Deep Learning Revolutionize Mobile Sensing?. In HotMobile '15.Google Scholar
- N. D. Lane, E. Miluzzo, H. Lu, D. Peebles, T. Choudhury, and A. T. Campbell. 2010. A survey of mobile phone sensing. IEEE Communications Magazine (2010).Google Scholar
- Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE (1998).Google Scholar
- En Li, Zhi Zhou, and Xu Chen. 2018. Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy. In MECOMM '18.Google ScholarDigital Library
- Z. Li and D. Hoiem. 2018. Learning without Forgetting. TPAMI (2018).Google Scholar
- TensorFlow Lite. 2020. Deploy machine learning models on mobile and IoT devices. https://www.tensorflow.org/lite.Google Scholar
- Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. 2017. Learning Efficient Convolutional Networks Through Network Slimming. In ICCV '17.Google Scholar
- Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019. Rethinking the Value of Network Pruning. In ICLR.Google Scholar
- Nicolas Meuleau, Emmanuel Benazera, Ronen I. Brafman, Eric A. Hansen, and Mausam. 2009. A Heuristic Search Approach to Planning with Continuous Resources in Stochastic Domains. J. Artif. Int. Res. (2009).Google Scholar
- Riccardo Miotto, Fei Wang, Shuang Wang, and Xiaoqian Jiang. 2017. Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics (2017).Google Scholar
- David C Mohr, Mi Zhang, and Stephen M Schueller. 2017. Personal Sensing: Understanding Mental Health Using Ubiquitous Sensors and Machine Learning. Annual review of clinical psychology (2017).Google Scholar
- Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning Convolutional Neural Networks for Resource Efficient Inference. In ICRL '17.Google Scholar
- Eugenio E. Müller, Vittorio Locatelli, and Daniela Cocchi. 1999. Neuroendocrine Control of Growth Hormone Secretion. Physiological Reviews (1999).Google Scholar
- Greig Paul and James Irvine. 2014. Privacy Implications of Wearable Health Devices. In SINCONF '14.Google Scholar
- Kaveena Persand, Andrew Anderson, and David Gregg. 2020. Composition of Saliency Metrics for Channel Pruning with a Myopic Oracle. (2020).Google Scholar
- Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V. Le. 2018. Regularized Evolution for Image Classifier Architecture Search. CoRR (2018).Google Scholar
- Sebastian Ruder. 2016. An overview of gradient descent optimization algorithms. CoRR (2016).Google Scholar
- Karen Simonyan and Andrew Zisserman. 2014. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv (2014).Google Scholar
- Tri Tran, Luke Marsh, and Robert Hunjet. 2019. Reinforcement Learning with Model Predictive Control - Recent Development.Google Scholar
- Terry T. Um, Franz M.J. Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. 2017. Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring Using Convolutional Neural Networks. In ICMI '17.Google ScholarDigital Library
- J. Wang, B. Cao, P. Yu, L. Sun, W. Bao, and X. Zhu. 2018. Deep Learning towards Mobile Applications. In ICDCS'18.Google Scholar
- Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from a Few Examples: A Survey on Few-Shot Learning. ACM Comput. Surv. (2020).Google ScholarDigital Library
- Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong. 2019. Federated Machine Learning: Concept and Applications. TIST (2019).Google ScholarDigital Library
- Shuochao Yao, Yiran Zhao, Huajie Shao, ShengZhong Liu, Dongxin Liu, Lu Su, and Tarek Abdelzaher. 2018. FastDeepIoT: Towards Understanding and Optimizing Neural Network Execution Time on Mobile and Embedded Devices. In SenSys '18.Google ScholarDigital Library
- Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. 2018. Lifelong Learning with Dynamically Expandable Networks. In ICLR '18.Google Scholar
- Zhiwen Yu, He Du, Fei Yi, Zhu Wang, and Bin Guo. 2019. Ten scientific problems in human behavior understanding. CCF TPCI (2019).Google Scholar
- Lei Zhang. 2019. Transfer Adaptation Learning: A Decade Survey. CoRR (2019).Google Scholar
- Yu Zhang, Tao Gu, Chu Luo, Vassilis Kostakos, and Aruna Seneviratne. 2018. FinDroidHR: Smartwatch Gesture Input with Optical Heartrate Monitor. IMWUT '18 (2018).Google ScholarDigital Library
- Yu Zhang, Tao Gu, and Xi Zhang. 2020. MDLdroid: a ChainSGD-reduce Approach to Mobile Deep Learning for Personal Mobile Sensing. In IPSN '20.Google ScholarCross Ref
- Michael Zhu and Suyog Gupta. 2018. To prune, or not to prune: exploring the efficacy of pruning for model compression. (2018).Google Scholar
- Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. 2017. Learning Transferable Architectures for Scalable Image Recognition. CoRR (2017).Google Scholar
Index Terms
MDLdroidLite: a release-and-inhibit control approach to resource-efficient deep neural networks on mobile devices
Recommendations
Mobile Sensing Through Deep Learning
Ph.D. Forum '17: Proceedings of the 2017 Workshop on MobiSys 2017 Ph.D. ForumToday, mobile devices are equipped with powerful processors along with various on-device sensors. Over the past few years, deep learning has become the dominant approach in the field of machine learning due to its impressive performance. We envision ...
ParallelFusion: Towards Maximum Utilization of Mobile GPU for DNN Inference
EMDL'21: Proceedings of the 5th International Workshop on Embedded and Mobile Deep LearningMobile GPUs are extremely under-utilized for DNN computations across different mobile deep learning frameworks and multiple DNNs with various complexities. We explore the feasibility of batching and it improves the throughput by up to 35%. However, real-...
Band: coordinated multi-DNN inference on heterogeneous mobile processors
MobiSys '22: Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and ServicesThe rapid development of deep learning algorithms, as well as innovative hardware advancements, encourages multi-DNN workloads such as augmented reality applications. However, existing mobile inference frameworks like TensorFlow Lite and MNN fail to ...
Comments