skip to main content
research-article
Public Access

ICNN: The Iterative Convolutional Neural Network

Authors Info & Claims
Published:14 December 2019Publication History
Skip Abstract Section

Abstract

Modern and recent architectures of vision-based Convolutional Neural Networks (CNN) have improved detection and prediction accuracy significantly. However, these algorithms are extremely computationally intensive. To break the power and performance wall of CNN computation, we reformulate the CNN computation into an iterative process, where each iteration processes a sub-sample of input features with smaller network and ingests additional features to improve the prediction accuracy. Each smaller network could either classify based on its input set or feed computed and extracted features to the next network to enhance the accuracy. The proposed approach allows early-termination upon reaching acceptable confidence. Moreover, each iteration provides a contextual awareness that allows an intelligent resource allocation and optimization for the proceeding iterations. In this article, we propose various policies to reduce the computational complexity of CNN through the proposed iterative approach. We illustrate how the proposed policies construct a dynamic architecture suitable for a wide range of applications with varied accuracy requirements, resources, and time-budget, without further need for network re-training. Furthermore, we carry out a visualization of the detected features in each iteration through deconvolution network to gain more insight into the successive traversal of the ICNN.

References

  1. Kyumin Lee, James Caverlee, and Steve Webb. 2010. Uncovering social spammers: Social honeypots+ machine learning. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 435--442.Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Min Chen, Yixue Hao, Kai Hwang, Lu Wang, and Lin Wang. 2017. Disease prediction by machine learning over big data from healthcare communities. IEEE Access 5 (2017), 8869--8879.Google ScholarGoogle ScholarCross RefCross Ref
  3. Marjan Saadati, Jill Nelson, and Hasan Ayaz. 2020. Multimodal fNIRS-EEG classification using deep-learning algorithms for brain-computer interfaces purposes. In Advances in Neuroergonomics and Cognitive Engineering, Hasan Ayaz (Ed.). Springer International Publishing, Cham, 209--220.Google ScholarGoogle Scholar
  4. Marjan Saadati, Jill Nelson, and Hasan Ayaz. 2020. Convolutional neural network for hybrid fNIRS-EEG mental workload classification. In Advances in Neuroergonomics and Cognitive Engineering, Hasan Ayaz (Ed.). Springer International Publishing, Cham, 221--232.Google ScholarGoogle Scholar
  5. Ivan Firdausi, Alva Erwin, Anto Satriyo Nugroho et al. 2010. Analysis of machine learning techniques used in behavior-based malware detection. In Proceedings of the 2nd International Conference on Advances in Computing, Control, and Telecommunication Technologies. IEEE, 201--203.Google ScholarGoogle Scholar
  6. Igor Santos, Jaime Devesa, Felix Brezo, Javier Nieves, and Pablo Garcia Bringas. 2013. Opem: A static-dynamic approach for machine-learning-based malware detection. In Proceedings of the International Joint Conference CISIS’12-ICEUTE’12-SOCO’12 Special Sessions. Springer, 271--280.Google ScholarGoogle ScholarCross RefCross Ref
  7. Hossein Sayadi, Hosein Mohammadi Makrani, Sai Manoj Pudukotai Dinakarrao, Tinoosh Mohsenin, Avesta Sasan, Setareh Rafatirad, and Houman Homayoun. 2019. 2SMaRT: A two-stage machine learning-based approach for run-time specialized hardware-assisted malware detection. In Proceedings of the Design, Automation 8 Test in Europe Conference 8 Exhibition (DATE’19). IEEE, 728--733.Google ScholarGoogle ScholarCross RefCross Ref
  8. D. Nemirovsky, T. Arkose, N. Markovic, M. Nemirovsky, O. Unsal, and A. Cristal. 2017. A machine learning approach for performance prediction and scheduling on heterogeneous CPUs. In Proceedings of the 29th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD’17). 121--128. DOI:http://dx.doi.org/10.1109/SBAC-PAD.2017.23Google ScholarGoogle Scholar
  9. Hosein Mohammadi Makrani, Hossein Sayadi, Tinoosh Mohsenin, Avesta Sasan, Houman Homayoun et al. 2019. XPPE: Cross-platform performance estimation of hardware accelerators using machine learning. In Proceedings of the 24th Asia and South Pacific Design Automation Conference. ACM, 727--732.Google ScholarGoogle Scholar
  10. Hossein Sayadi, Nisarg Patel, Avesta Sasan, and Houman Homayoun. 2017. Machine learning-based approaches for energy-efficiency prediction and scheduling in composite cores architectures. In Proceedings of the IEEE International Conference on Computer Design (ICCD’17). IEEE, 129--136.Google ScholarGoogle ScholarCross RefCross Ref
  11. Raymond Sepe Jr. 2005. Voice actuation with contextual learning for intelligent machine control. U.S. Patent 6,895,380.Google ScholarGoogle Scholar
  12. Charu C. Aggarwal and ChengXiang Zhai. 2012. Mining Text Data. Springer Science 8 Business Media.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436.Google ScholarGoogle Scholar
  14. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25. MIT Press, 1097--1105.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).Google ScholarGoogle Scholar
  16. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. 2016. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2921--2929.Google ScholarGoogle ScholarCross RefCross Ref
  17. Christian Szegedy et al. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--9.Google ScholarGoogle Scholar
  18. Kaiming He et al. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.Google ScholarGoogle Scholar
  19. Katayoun Neshatpour, Avesta Sasan, and Houman Homayoun. 2016. Big data analytics on heterogeneous accelerator architectures. In Proceedings of the International Conference on Hardware/Software Codesign and System Synthesis (CODES+ ISSS’16). IEEE, 1--3.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Katayoun Neshatpour, Arezou Koohi, Farnoud Farahmand, Rajiv Joshi, Setareh Rafatirad, Avesta Sasan, and Houman Homayoun. 2016. Big biomedical image processing hardware acceleration: A case study for K-means and image filtering. In Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS’16). IEEE, 1134--1137.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Katayoun Neshatpour, Farnaz Behnia, Houman Homayoun, and Avesta Sasan. 2018. ICNN: An iterative implementation of convolutional neural networks to enable energy and computational complexity aware dynamic approximation. In Proceedings of the Design Automation and Test in Europe.Google ScholarGoogle ScholarCross RefCross Ref
  22. Mengchen Liu et al. 2017. Towards better analysis of deep convolutional neural networks. IEEE Trans. Visual. Comput. Graph. 23, 1 (2017), 91--100.Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Yann LeCun et al. 1989. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1, 4 (1989), 541--551.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Yu-Hsin Chen et al. 2017. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. IEEE J. Solid-State Circ. 52, 1 (2017), 127--138.Google ScholarGoogle ScholarCross RefCross Ref
  25. J. Deng et al. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR’09).Google ScholarGoogle ScholarCross RefCross Ref
  26. Kaiming He et al. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision. 1026--1034.Google ScholarGoogle Scholar
  27. Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2818--2826.Google ScholarGoogle ScholarCross RefCross Ref
  28. Christopher M. Bishop. 1995. Neural Networks for Pattern Recognition. Oxford University Press.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Brian D. Ripley. 2007. Pattern Recognition and Neural Networks. Cambridge University Press.Google ScholarGoogle Scholar
  30. A. Jafari, A. Ganesan, C. S. K. Thalisetty, V. Sivasubramanian, T. Oates, and T. Mohsenin. 2019. SensorNet: A scalable and low-power deep convolutional neural network for multimodal data classification. IEEE Trans. Circ. Syst. I: Regular Papers 66, 1 (Jan. 2019), 274--287. DOI:http://dx.doi.org/10.1109/TCSI.2018.2848647Google ScholarGoogle Scholar
  31. Morteza Hosseini, Mark Horton, Hiren Paneliya, Uttej Kallakuri, Houman Homayoun, and Tinoosh Mohsenin. 2019. On the complexity reduction of dense layers from O(N2) to O(NlogN) with cyclic sparsely connected layers. In Proceedings of the 56th Annual Design Automation Conference. ACM.Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Bharat Prakash, Mark Horton, Nicholas Waytowich, William David Hairston, Tim Oates, and Tinoosh Mohsenin. 2019. On the use of deep autoencoders for efficient embedded reinforcement learning. In Proceedings of the 29th Edition of the Great Lakes Symposium on VLSI (GLSVLSI’19). ACM.Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. 2014. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.3866 (2014).Google ScholarGoogle Scholar
  34. Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. 2014. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv preprint arXiv:1412.6553 (2014).Google ScholarGoogle Scholar
  35. Baoyuan Liu, Min Wang, Hassan Foroosh, Marshall Tappen, and Marianna Pensky. 2015. Sparse convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 806--814.Google ScholarGoogle Scholar
  36. Morteza Hosseini, Hiren Paneliya, Mohit Kallakuri, Uttej Khatwani, and Tinoosh Mohsenin. 2019. Minimizing classification energy of binarized neural network inference for wearable devices. In Proceedings of the 20th International Symposium on Quality Electronic Design (ISQED’19). IEEE.Google ScholarGoogle ScholarCross RefCross Ref
  37. Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Conference on Computer Vision (ECCV’16). Springer, 525--542.Google ScholarGoogle ScholarCross RefCross Ref
  38. Shimeng Yu, Zhiwei Li, Pai-Yu Chen, Huaqiang Wu, Bin Gao, Deli Wang, Wei Wu, and He Qian. 2016. Binary neural network with 16 Mb RRAM macro chip for classification and online training. In Proceedings of the IEEE International Electron Devices Meeting (IEDM’16). IEEE, 16--2.Google ScholarGoogle ScholarCross RefCross Ref
  39. Jung H. Kim and Sung-Kwon Park. 1995. The geometrical learning of binary neural networks. IEEE Trans Neural Netw. 6, 1 (1995), 237--247.Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks. In Advances in Neural Information Processing Systems. MIT Press, 4107--4115.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 (2016).Google ScholarGoogle Scholar
  42. Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. 2016. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4820--4828.Google ScholarGoogle ScholarCross RefCross Ref
  43. Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. 2016. DoReFa-Net: Training low bitwidth convolutional neural networks with low bit-width gradients. arXiv preprint arXiv:1606.06160 (2016).Google ScholarGoogle Scholar
  44. Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. 2014. Low precision arithmetic for deep learning. CoRR abs/1412.7024 4 (2014).Google ScholarGoogle Scholar
  45. Suyog Gupta, Ankur Agrawal, and Pritish Narayanan. 2015. Deep learning with limited numerical precision. In Proceedings of the International Conference on Machine Learning (ICML’15). 1737--1746.Google ScholarGoogle Scholar
  46. Darryl Dexu Lin, Sachin S. Talathi, and V. Sreekanth Annapureddy. 2016. Fixed point quantization of deep convolutional networks. In Proceedings of the International Conference on Machine Learning (ICML’16). 2849--2858.Google ScholarGoogle Scholar
  47. Yu-Hsin Chen, Joel Emer, and Vivienne Sze. 2016. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. In ACM SIGARCH Computer Architecture News, Vol. 44. IEEE Press, 367--379.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Tianshi Chen, Zidong Du, Ninghui Sun, Jia Wang, Chengyong Wu, Yunji Chen, and Olivier Temam. 2014. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In ACM Sigplan Notices, Vol. 49. ACM, 269--284.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Zidong Du, Robert Fasthuber, Tianshi Chen, Paolo Ienne, Ling Li, Tao Luo, Xiaobing Feng, Yunji Chen, and Olivier Temam. 2015. ShiDianNao: Shifting vision processing closer to the sensor. In ACM SIGARCH Computer Architecture News, Vol. 43. ACM, 92--104.Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Kamel Abdelouahab, Maxime Pelcat, Jocelyn Serot, and François Berry. 2018. Accelerating CNN inference on FPGAs: A survey. arXiv preprint arXiv:1806.01683 (2018).Google ScholarGoogle Scholar
  51. Katayoun Neshatpour, Hosein Mohammadi Mokrani, Avesta Sasan, Hassan Ghasemzadeh, Setareh Rafatirad, and Houman Homayoun. 2018. Architectural considerations for FPGA acceleration of machine learning applications in MapReduce. In Proceedings of the 18th International Conference on Embedded Computer Systems: Architectures, Modeling, and Simulation. ACM, 89--96.Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. Chen Zhang, Di Wu, Jiayu Sun, Guangyu Sun, Guojie Luo, and Jason Cong. 2016. Energy-efficient CNN implementation on a deeply pipelined FPGA cluster. In Proceedings of International Symposium on Low-Power Electronics and Design. 326--331.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Katayoun Neshatpour, Hosein Mohammadi Makrani, Avesta Sasan, Hassan Ghasemzadeh, Setareh Rafatirad, and Houman Homayoun. 2018. Design space exploration for hardware acceleration of machine learning applications in MapReduce. In Proceedings of the IEEE 26th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM’18). IEEE, 221--221.Google ScholarGoogle ScholarCross RefCross Ref
  54. Hokchhay Tann, Soheil Hashemi, R. Iris Bahar, and Sherief Reda. 2016. Runtime configurable deep neural networks for energy-accuracy trade-off. In Proceedings of the International Conference on Hardware/Software Codesign and System Synthesis (CODES+ ISSS’16). IEEE, 1--10.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. Priyadarshini Panda, Abhronil Sengupta, and Kaushik Roy. 2016. Conditional deep learning for energy-efficient and enhanced pattern recognition. In Proceedings of the Design, Automation 8 Test in Europe Conference. IEEE, 475--480.Google ScholarGoogle ScholarCross RefCross Ref
  56. Surat Teerapittayanon, Bradley McDanel, and HT Kung. 2016. Branchynet: Fast inference via early exiting from deep neural networks. In Proceedings of the International Conference on Pattern Recognition (ICPR’16). IEEE, 2464--2469.Google ScholarGoogle ScholarCross RefCross Ref
  57. Li Deng. 2012. The MNIST database of handwritten digit images for machine learning research [best of the web]. IEEE Signal Process. Mag. 29, 6 (2012), 141--142.Google ScholarGoogle ScholarCross RefCross Ref
  58. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. Understanding the exploding gradient problem. CoRR abs/1211.5063 (2012).Google ScholarGoogle Scholar
  59. Katayoun Neshatpour, Farnaz Behnia, Houman Homayoun, and Avesta Sasan. 2019. Exploiting energy-accuracy trade-off through contextual awareness in multi-stage convolutional neural networks. In Proceedings of the 20th International Symposium on Quality Electronic Design (ISQED’19). IEEE, 265--270.Google ScholarGoogle ScholarCross RefCross Ref
  60. C. Sidney Burrus et al. 1997. Introduction to wavelets and wavelet transforms: A primer. (1997).Google ScholarGoogle Scholar
  61. Yangqing Jia et al. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM International Conference on Multimedia. ACM, 675--678.Google ScholarGoogle Scholar
  62. Stéphane Mallat. 1999. A Wavelet Tour of Signal Processing. Academic Press.Google ScholarGoogle Scholar
  63. Kamrul Hasan Talukder and Koichi Harada. 2010. Haar wavelet-based approach for image compression and quality assessment of compressed image. arXiv preprint arXiv:1010.4084 (2010).Google ScholarGoogle Scholar
  64. V. Elamaran and A. Praveen. 2012. Comparison of DCT and wavelets in image coding. In Proceedings of the International Conference on Computer Communication and Informatics. 1--4.Google ScholarGoogle Scholar
  65. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2014. Spatial pyramid pooling in deep convolutional networks for visual recognition. In Proceedings of the European Conference on Computer Vision. Springer, 346--361.Google ScholarGoogle ScholarCross RefCross Ref
  66. Matthew D. Zeiler, Graham W. Taylor, and Rob Fergus. 2011. Adaptive deconvolutional networks for mid and high level feature learning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’11). IEEE, 2018--2025.Google ScholarGoogle ScholarDigital LibraryDigital Library
  67. Wenzhe Shi, Jose Caballero, Lucas Theis, Ferenc Huszar, Andrew Aitken, Christian Ledig, and Zehan Wang. 2016. Is the deconvolution layer the same as a convolutional layer? arXiv preprint arXiv:1609.07009 (2016).Google ScholarGoogle Scholar
  68. Matthew D. Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In Proceedings of the European Conference on Computer Vision. Springer, 818--833.Google ScholarGoogle Scholar
  69. Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3431--3440.Google ScholarGoogle ScholarCross RefCross Ref
  70. Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, and Thomas Brox. 2015. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision. 2758--2766.Google ScholarGoogle Scholar
  71. Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 (2015).Google ScholarGoogle Scholar
  72. Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. 2017. Structured pruning of deep convolutional neural networks. J. Emerg. Technol. Comput. Syst. 13, 3, Article 32 (Feb. 2017), 18 pages. DOI:http://dx.doi.org/10.1145/3005348Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).Google ScholarGoogle ScholarDigital LibraryDigital Library
  74. Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark A. Horowitz, and William J. Dally. 2016. EIE: Efficient inference engine on compressed deep neural network. In Proceedings of the 43rd International Symposium on Computer Architecture. IEEE Press, 243--254.Google ScholarGoogle Scholar
  75. Song Han, Huizi Mao, and William J. Dally. 2015. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015).Google ScholarGoogle ScholarDigital LibraryDigital Library
  76. Andrew G. Howard, Menglong Zhu, Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).Google ScholarGoogle Scholar
  77. Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4510--4520.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. ICNN: The Iterative Convolutional Neural Network

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!