skip to main content
research-article

Neural Network Pruning by Recurrent Weights for Finance Market

Authors Info & Claims
Published:22 January 2022Publication History
Skip Abstract Section

Abstract

Convolutional Neural Networks (CNNs) and deep learning technology are applied in current financial market to rapidly promote the development of finance market and Internet economy. The continuous development of neural networks with more hidden layers improves the performance but increases the computational complexity. Generally, channel pruning methods are useful to compact neural networks. However, typical channel pruning methods would remove layers by mistake due to the static pruning ratio of manual setting, which could destroy the whole structure of neural networks. It is difficult to improve the ratio of compressing neural networks only by pruning channels while maintaining good network structures.

Therefore, we propose a novel neural Networks Pruning by Recurrent Weights (NPRW) that can repeatedly evaluate the significance of weights and adaptively adjust them to compress neural networks within acceptable loss of accuracy. The recurrent weights with low sensitivity are compulsorily set to zero by evaluating the magnitude of weights, and pruned network only uses a few significant weights. Then, we add the regularization to the scaling factors on neural networks, in which recurrent weights with high sensitivity can be dynamically updated and weights of low sensitivity stay at zero invariably. By this way, the significance of channels can be quantitatively evaluated by recurrent weights. It has been verified with typical neural networks of LeNet, VGGNet, and ResNet on multiple benchmark datasets involving stock index futures, digital recognition, and image classification. The pruned LeNet-5 achieves the 58.9% reduction amount of parameters with 0.29% loss of total accuracy for Shanghai and Shenzhen 300 stock index futures. As for the CIFAR-10, the pruned VGG-19 reduces more than 50% FLOPs, and the decrease of network accuracy is less than 0.5%. In addition, the pruned ResNet-164 tested on the SVHN reduces more than 58% FLOPs with relative improvement on accuracy by 0.11%.

REFERENCES

  1. [1] Chen C. H.. 1994. Neural networks for financial market prediction. In Proceedings of the IEEE International Conference on Neural Networks (ICNN). 11991202. DOI: 10.1109/ICNN.1994.374354Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Qi Songqiao, Jin Kaijun, Li Baisong, and Qian Yufeng. 2020. The exploration of internet finance by using neural network. J. Comput. Appl. Math. 369 (May 2020), 112630. DOI: 10.1016/j.cam.2019.112630Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. [3] Gómez Jon Ander, Arévalo Juan, Paredes Roberto, and Nin Jordi. 2018. End-to-end neural network architecture for fraud scoring in card payments. Pattern Recog. Lett. 105 (Aug. 2018), 175181. DOI: 10.1016/j.patrec.2017.08.024 Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. [4] Ryman-Tubb Nick F., Krause Paul, and Garn Wolfgang. 2018. How artificial intelligence and machine learning research impacts payment card fraud detection: A survey and industry benchmark. Eng. Applic. Artif. Intell. 76, NOV. (Sept. 2018), 130157. DOI: 10.1016/j.engappai.2018.07.008Google ScholarGoogle ScholarCross RefCross Ref
  5. [5] Chen Chunchun, Zhang Pu, Liu Yuan, and Liu Jun. 2020. Financial quantitative investment using convolutional neural network and deep learning technology. Neurocomputing (May 2020), 384390. DOI: 10.1016/j.neucom.2019.09.092Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Zhang Jiboning. 2020. Investment risk model based on intelligent fuzzy neural network and VaR. J. Comput. Appl. Math. 371 (June 2020), 112707. DOI: 10.1016/j.cam.2019.112707Google ScholarGoogle ScholarCross RefCross Ref
  7. [7] Li He, Ota Kaoru, and Dong Mianxiong. 2018. Learning IoT in edge: Deep learning for the internet of things with edge computing. IEEE Netw. 32, 1 (Jan. 2018), 96101. DOI: 10.1109/MNET.2018.1700202Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Tsantekidis Avraam, Passalis Nikolaos, Tefas Anastasios, Kanniainen Juho, and Iosifidis Alexandros. 2017. Forecasting stock prices from the limit order book using convolutional neural networks. In Proceedings of the IEEE 19th Conference on Business Informatics (CBI). 712. DOI: 10.1109/CBI.2017.23Google ScholarGoogle ScholarCross RefCross Ref
  9. [9] Schroff Florian, Kalenichenko Dmitry, and Philbin James. 2015. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 815823. DOI: 10.1109/CVPR.2015.7298682Google ScholarGoogle ScholarCross RefCross Ref
  10. [10] Bhunia Ayan Kumar, Das Abhirup, Bhunia Ankan Kumar, Kishore Perla Sai Raj, and Roy Partha Pratim. 2019. Handwriting recognition in low-resource scripts using adversarial learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 47674776. DOI: 10.1109/CVPR.2019.00490Google ScholarGoogle ScholarCross RefCross Ref
  11. [11] Krizhevsky Alex, Sutskever Ilya, and Hinton Geoffrey E.. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 10971105. DOI: 4824-imagenet-classification-with-deep-convolutional-neural-networks. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Srivastava Rupesh K., Greff Klaus, and Schmidhuber Jürgen. 2015. Training very deep networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 23772385. DOI: 5850-training-very-deep-networks. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Pei Songwen., Tang Fuwu., Ji Yanfei., Fan Jing., and Ning Zhong.. 2018. Localized traffic sign detection with multi-scale deconvolution networks. In Proceedings of the IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). 355360. DOI: 10.1109/COMPSAC.2018.00056Google ScholarGoogle ScholarCross RefCross Ref
  14. [14] Ye Jianbo, Lu Xin, Lin Zhe, and Wang James Z.. 2018. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. In Proceedings of the International Conference on Learning Representations (ICLR). 111. DOI: 1802.00124Google ScholarGoogle Scholar
  15. [15] Pei Songwen, Shen Tianma, Wang Xianrong, Gu Chunhua, Ning Zhong, Ye Xiaochun, and Xiong Naixue. 2020. 3DACN: 3D augmented convolutional network for time series data. Inf. Sci. (Mar. 2020), 1729. DOI: 10.1016/j.ins.2019.11.040Google ScholarGoogle ScholarCross RefCross Ref
  16. [16] Ma Ningning, Zhang Xiangyu, Zheng Hai-Tao, and Sun Jian. 2018. Shufflenet v2: Practical guidelines for efficient CNN architecture design. In Proceedings of the European Conference on Computer Vision (ECCV). 122138. DOI: 10.1007/978-3-030-01264-98Google ScholarGoogle ScholarCross RefCross Ref
  17. [17] Sandler Mark, Howard Andrew, Zhu Menglong, Zhmoginov Andrey, and Chen Liang-Chieh. 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 45104520. DOI: 10.1109/CVPR.2018.00474Google ScholarGoogle ScholarCross RefCross Ref
  18. [18] Rastegari Mohammad, Ordonez Vicente, Redmon Joseph, and Farhadi Ali. 2016. XNOR-Net: ImageNet classification using binary convolutional neural networks. In Proceedings of the European Conference on Computer Vision (ECCV). 525542. DOI: 10.1007/978-3-319-46493-032Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Denil Misha, Shakibi Babak, Dinh Laurent, Ranzato Marc’Aurelio, and Freitas Nando De. 2013. Predicting parameters in deep learning. In Proceedings of the 26th International Conference on Neural Information Processing System (NIPS). 21482156. DOI: 10.5555/2999792.2999852 Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Li Hao, Kadav Asim, Durdanovic Igor, Samet Hanan, and Graf Hans Peter. 2016. Pruning filters for efficient ConvNets. In Proceedings of the International Conference on Learning Representations (ICLR). 113. DOI: 1608.08710Google ScholarGoogle Scholar
  21. [21] Hu Hengyuan, Peng Rui, Tai Yu-Wing, and Tang Chi-Keung. 2017. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. In Proceedings of the International Conference on Learning Representations (ICLR). 19. DOI: 1607.03250Google ScholarGoogle Scholar
  22. [22] Wen Wei, Wu Chunpeng, Wang Yandan, Chen Yiran, and Li Hai. 2016. Learning structured sparsity in deep neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 20742082. DOI: 6504-learning-structured-sparsity-in-deep-neural-networks. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. [23] Alvarez Jose M. and Salzmann Mathieu. 2016. Learning the number of neurons in deep networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 22702278. DOI: 6372-learning-the-number-of-neurons-in-deep-networks. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] He Yihui, Zhang Xiangyu, and Sun Jian. 2017. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 13981406. DOI: 10.1109/ICCV.2017.155Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Zhuang Zhuangwei, Tan Mingkui, Zhuang Bohan, Liu Jing, Guo Yong, Wu Qingyao, Huang Junzhou, and Zhu Jinhui. 2018. Discrimination-aware channel pruning for deep neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 875886. DOI: 7367-discrimination-aware-channel-pruning-for-deep-neural-networks. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Han Song, Pool Jeff, Tran John, and Dally William. 2015. Learning both weights and connections for efficient neural network. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 11351143. DOI: 5784-learning-both-weights-and-connections-for-efficient-neural-network. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. [27] Han Song, Mao Huizi, and Dally W. J.. 2016. Compressing deep neural networks with pruning trained quantization and Huffman coding. In Proceedings of the International Conference on Learning Representations (ICLR). 114. DOI: 1510.00149Google ScholarGoogle Scholar
  28. [28] Pei Songwen, Wu Yusheng, and Qiu Meikang. 2020. Neural network compression and acceleration by federated pruning. In Proceedings of the 20th International Conference on Algorithm and Architecture for Parallel Processing (ICA3PP). 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Denton Emily, Zaremba Wojciech, Bruna Joan, LeCun Yann, and Fergus Rob. 2014. Exploiting linear structure within convolutional networks for efficient evaluation. In Proceedings of the 27th International Conference on Neural Information Processing Systems. 12691277. DOI: 5544-exploiting-linear-structure-within-convolutional-networks-for-efficient-evaluation. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. [30] Wang Min, Liu Baoyuan, and Foroosh Hassan. 2016. Design of efficient convolutional layers using single intra-channel convolution, topological subdivisioning and spatial “bottleneck” structure. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 19. DOI: 1608.04337Google ScholarGoogle Scholar
  31. [31] Hou Lu and Kwok James T.. 2018. Loss-aware weight quantization of deep networks. In Proceedings of the International Conference on Learning Representations (ICLR). 116. DOI: 1802.08635Google ScholarGoogle Scholar
  32. [32] Luo Jian-Hao, Wu Jianxin, and Lin Weiyao. 2017. ThiNet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 50685076. DOI: 10.1109/ICCV.2017.541Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Chen Wenlin, Wilson James T., Tyree Stephen, Weinberger Kilian Q., and Chen Yixin. 2015. Compressing neural networks with the hashing trick. In Proceedings of the 32nd International Conference on International Conference on Machine Learning. 22852294. DOI: 10.5555/3045118.3045361 Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Liu Zhuang, Li Jianguo, Shen Zhiqiang, Huang Gao, Yan Shoumeng, and Zhang Changshui. 2017. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 27552763. DOI: 10.1109/ICCV.2017.298Google ScholarGoogle ScholarCross RefCross Ref
  35. [35] Qiu Meikang and Sha Edwin H.-M.. 2009. Cost minimization while satisfying hard/soft timing constraints for heterogeneous embedded systems. ACM Trans. Des. Autom. Electron. Syst. 14, 2 (2009), 130. DOI: 10.1145/1497561.1497568 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Qiu Meikang, Chen Zhi, Niu Jianwei, Zong Ziliang, Quan Gang, Qin Xiao, and Yang Laurence T.. 2015. Data allocation for hybrid memory with genetic algorithm. IEEE Trans. Emerg. Topics Comput. 3, 4 (Feb. 2015), 544555. DOI: 10.1109/TETC.2015.2398824 Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. [37] Qiu Han, Qiu Meikang, Lu Zhihui, and Memmi Gerard. 2019. An efficient key distribution system for data fusion in V2X heterogeneous networks. Inf. Fusion (Feb. 2019). DOI: 10.1016/j.inffus.2019.02.002Google ScholarGoogle ScholarCross RefCross Ref
  38. [38] He K., Zhang X., Ren S., and Sun J.. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770778. DOI: 10.1109/CVPR.2016.90Google ScholarGoogle ScholarCross RefCross Ref
  39. [39] Reed R.. 1993. Pruning algorithms—A survey. IEEE Trans. Neural Netw. 4, 5 (Sep. 1993), 740747. DOI: 10.1109/72.248452 Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. [40] Simonyan Karen and Zisserman Andrew. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR). 114. DOI: 1409.1556Google ScholarGoogle Scholar
  41. [41] Huang Gao, Sun Yu, Liu Zhuang, Sedra Daniel, and Weinberger Kilian Q.. 2016. Deep networks with stochastic depth. In Proceedings of the European Conference on Computer Vision (ECCV). 646661. DOI: 10.1007/978-3-319-46493-039Google ScholarGoogle ScholarCross RefCross Ref
  42. [42] He Kaiming, Zhang Xiangyu, Ren Shaoqing, and Sun Jian. 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 10261034. DOI: 10.1109/ICCV.2015.123Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Neural Network Pruning by Recurrent Weights for Finance Market

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Internet Technology
      ACM Transactions on Internet Technology  Volume 22, Issue 3
      August 2022
      631 pages
      ISSN:1533-5399
      EISSN:1557-6051
      DOI:10.1145/3498359
      • Editor:
      • Ling Liu
      Issue’s Table of Contents

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 22 January 2022
      • Accepted: 1 November 2021
      • Revised: 1 October 2020
      • Received: 1 August 2020
      Published in toit Volume 22, Issue 3

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Full Text

    View this article in Full Text.

    View Full Text

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!