Abstract
Convolutional Neural Networks (CNNs) and deep learning technology are applied in current financial market to rapidly promote the development of finance market and Internet economy. The continuous development of neural networks with more hidden layers improves the performance but increases the computational complexity. Generally, channel pruning methods are useful to compact neural networks. However, typical channel pruning methods would remove layers by mistake due to the static pruning ratio of manual setting, which could destroy the whole structure of neural networks. It is difficult to improve the ratio of compressing neural networks only by pruning channels while maintaining good network structures.
Therefore, we propose a novel neural Networks Pruning by Recurrent Weights (NPRW) that can repeatedly evaluate the significance of weights and adaptively adjust them to compress neural networks within acceptable loss of accuracy. The recurrent weights with low sensitivity are compulsorily set to zero by evaluating the magnitude of weights, and pruned network only uses a few significant weights. Then, we add the regularization to the scaling factors on neural networks, in which recurrent weights with high sensitivity can be dynamically updated and weights of low sensitivity stay at zero invariably. By this way, the significance of channels can be quantitatively evaluated by recurrent weights. It has been verified with typical neural networks of LeNet, VGGNet, and ResNet on multiple benchmark datasets involving stock index futures, digital recognition, and image classification. The pruned LeNet-5 achieves the 58.9% reduction amount of parameters with 0.29% loss of total accuracy for Shanghai and Shenzhen 300 stock index futures. As for the CIFAR-10, the pruned VGG-19 reduces more than 50% FLOPs, and the decrease of network accuracy is less than 0.5%. In addition, the pruned ResNet-164 tested on the SVHN reduces more than 58% FLOPs with relative improvement on accuracy by 0.11%.
- [1] . 1994. Neural networks for financial market prediction. In Proceedings of the IEEE International Conference on Neural Networks (ICNN). 1199–1202.
DOI: 10.1109/ICNN.1994.374354Google ScholarCross Ref
- [2] . 2020. The exploration of internet finance by using neural network. J. Comput. Appl. Math. 369 (
May 2020), 112630.DOI: 10.1016/j.cam.2019.112630Google ScholarDigital Library
- [3] . 2018. End-to-end neural network architecture for fraud scoring in card payments. Pattern Recog. Lett. 105 (
Aug. 2018), 175–181.DOI: 10.1016/j.patrec.2017.08.024 Google ScholarDigital Library
- [4] . 2018. How artificial intelligence and machine learning research impacts payment card fraud detection: A survey and industry benchmark. Eng. Applic. Artif. Intell. 76, NOV. (
Sept. 2018), 130–157.DOI: 10.1016/j.engappai.2018.07.008Google ScholarCross Ref
- [5] . 2020. Financial quantitative investment using convolutional neural network and deep learning technology. Neurocomputing (
May 2020), 384–390.DOI: 10.1016/j.neucom.2019.09.092Google ScholarCross Ref
- [6] . 2020. Investment risk model based on intelligent fuzzy neural network and VaR. J. Comput. Appl. Math. 371 (
June 2020), 112707.DOI: 10.1016/j.cam.2019.112707Google ScholarCross Ref
- [7] . 2018. Learning IoT in edge: Deep learning for the internet of things with edge computing. IEEE Netw. 32, 1 (
Jan. 2018), 96–101.DOI: 10.1109/MNET.2018.1700202Google ScholarCross Ref
- [8] . 2017. Forecasting stock prices from the limit order book using convolutional neural networks. In Proceedings of the IEEE 19th Conference on Business Informatics (CBI). 7–12.
DOI: 10.1109/CBI.2017.23Google ScholarCross Ref
- [9] . 2015. FaceNet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 815–823.
DOI: 10.1109/CVPR.2015.7298682Google ScholarCross Ref
- [10] . 2019. Handwriting recognition in low-resource scripts using adversarial learning. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). 4767–4776.
DOI: 10.1109/CVPR.2019.00490Google ScholarCross Ref
- [11] . 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 1097–1105.
DOI: 4824-imagenet-classification-with-deep-convolutional-neural-networks. Google ScholarDigital Library
- [12] . 2015. Training very deep networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2377–2385.
DOI: 5850-training-very-deep-networks. Google ScholarDigital Library
- [13] . 2018. Localized traffic sign detection with multi-scale deconvolution networks. In Proceedings of the IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC). 355–360.
DOI: 10.1109/COMPSAC.2018.00056Google ScholarCross Ref
- [14] . 2018. Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. In Proceedings of the International Conference on Learning Representations (ICLR). 1–11.
DOI: 1802.00124Google Scholar - [15] . 2020. 3DACN: 3D augmented convolutional network for time series data. Inf. Sci. (
Mar. 2020), 17–29.DOI: 10.1016/j.ins.2019.11.040Google ScholarCross Ref
- [16] . 2018. Shufflenet v2: Practical guidelines for efficient CNN architecture design. In Proceedings of the European Conference on Computer Vision (ECCV). 122–138.
DOI: 10.1007/978-3-030-01264-98Google ScholarCross Ref
- [17] . 2018. MobileNetV2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 4510–4520.
DOI: 10.1109/CVPR.2018.00474Google ScholarCross Ref
- [18] . 2016. XNOR-Net: ImageNet classification using binary convolutional neural networks. In Proceedings of the European Conference on Computer Vision (ECCV). 525–542.
DOI: 10.1007/978-3-319-46493-032Google ScholarCross Ref
- [19] . 2013. Predicting parameters in deep learning. In Proceedings of the 26th International Conference on Neural Information Processing System (NIPS). 2148–2156.
DOI: 10.5555/2999792.2999852 Google ScholarDigital Library
- [20] . 2016. Pruning filters for efficient ConvNets. In Proceedings of the International Conference on Learning Representations (ICLR). 1–13.
DOI: 1608.08710Google Scholar - [21] . 2017. Network trimming: A data-driven neuron pruning approach towards efficient deep architectures. In Proceedings of the International Conference on Learning Representations (ICLR). 1–9.
DOI: 1607.03250Google Scholar - [22] . 2016. Learning structured sparsity in deep neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 2074–2082.
DOI: 6504-learning-structured-sparsity-in-deep-neural-networks. Google ScholarDigital Library
- [23] . 2016. Learning the number of neurons in deep networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 2270–2278.
DOI: 6372-learning-the-number-of-neurons-in-deep-networks. Google ScholarDigital Library
- [24] . 2017. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 1398–1406.
DOI: 10.1109/ICCV.2017.155Google ScholarCross Ref
- [25] . 2018. Discrimination-aware channel pruning for deep neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 875–886.
DOI: 7367-discrimination-aware-channel-pruning-for-deep-neural-networks. Google ScholarDigital Library
- [26] . 2015. Learning both weights and connections for efficient neural network. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 1135–1143.
DOI: 5784-learning-both-weights-and-connections-for-efficient-neural-network. Google ScholarDigital Library
- [27] . 2016. Compressing deep neural networks with pruning trained quantization and Huffman coding. In Proceedings of the International Conference on Learning Representations (ICLR). 1–14.
DOI: 1510.00149Google Scholar - [28] . 2020. Neural network compression and acceleration by federated pruning. In Proceedings of the 20th International Conference on Algorithm and Architecture for Parallel Processing (ICA3PP). 1–10.Google Scholar
Digital Library
- [29] . 2014. Exploiting linear structure within convolutional networks for efficient evaluation. In Proceedings of the 27th International Conference on Neural Information Processing Systems. 1269–1277.
DOI: 5544-exploiting-linear-structure-within-convolutional-networks-for-efficient-evaluation. Google ScholarDigital Library
- [30] . 2016. Design of efficient convolutional layers using single intra-channel convolution, topological subdivisioning and spatial “bottleneck” structure. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1–9.
DOI: 1608.04337Google Scholar - [31] . 2018. Loss-aware weight quantization of deep networks. In Proceedings of the International Conference on Learning Representations (ICLR). 1–16.
DOI: 1802.08635Google Scholar - [32] . 2017. ThiNet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 5068–5076.
DOI: 10.1109/ICCV.2017.541Google ScholarCross Ref
- [33] . 2015. Compressing neural networks with the hashing trick. In Proceedings of the 32nd International Conference on International Conference on Machine Learning. 2285–2294.
DOI: 10.5555/3045118.3045361 Google ScholarDigital Library
- [34] . 2017. Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2755–2763.
DOI: 10.1109/ICCV.2017.298Google ScholarCross Ref
- [35] . 2009. Cost minimization while satisfying hard/soft timing constraints for heterogeneous embedded systems. ACM Trans. Des. Autom. Electron. Syst. 14, 2 (2009), 1–30.
DOI: 10.1145/1497561.1497568 Google ScholarDigital Library
- [36] . 2015. Data allocation for hybrid memory with genetic algorithm. IEEE Trans. Emerg. Topics Comput. 3, 4 (
Feb. 2015), 544–555.DOI: 10.1109/TETC.2015.2398824 Google ScholarDigital Library
- [37] . 2019. An efficient key distribution system for data fusion in V2X heterogeneous networks. Inf. Fusion (
Feb. 2019).DOI: 10.1016/j.inffus.2019.02.002Google ScholarCross Ref
- [38] . 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 770–778.
DOI: 10.1109/CVPR.2016.90Google ScholarCross Ref
- [39] . 1993. Pruning algorithms—A survey. IEEE Trans. Neural Netw. 4, 5 (
Sep. 1993), 740–747.DOI: 10.1109/72.248452 Google ScholarDigital Library
- [40] . 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR). 1–14.
DOI: 1409.1556Google Scholar - [41] . 2016. Deep networks with stochastic depth. In Proceedings of the European Conference on Computer Vision (ECCV). 646–661.
DOI: 10.1007/978-3-319-46493-039Google ScholarCross Ref
- [42] . 2015. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 1026–1034.
DOI: 10.1109/ICCV.2015.123Google ScholarDigital Library
Index Terms
Neural Network Pruning by Recurrent Weights for Finance Market
Recommendations
A New Method to Prune the Neural Network
IJCNN '00: Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks (IJCNN'00)-Volume 6 - Volume 6Using backpropagation algorithm (BP) to train neural networks is a widely adopted practice in both theory and practical applications. But its distributed weight representation, that is the weight matrix of final network after training by using BP are ...
HFPQ: deep neural network compression by hardware-friendly pruning-quantization
AbstractThis paper presents a hardware-friendly compression method for deep neural networks. This method effectively combines layered channel pruning with quantization by a power exponential of 2. While keeping a small decrease in the accuracy of the ...






Comments