Abstract
The Internet of Things and big data are currently hot concepts and research fields. The mining, classification, and recognition of big data in the Internet of Things system are the key links that are widely of concern at present. The artificial neural network is beneficial for multi-dimensional data classification and recognition because of its strong feature extraction and self-learning ability. Pre-training is an effective method to address the gradient diffusion problem in deep neural networks and could result in better generalization. This article focuses on the performance of supervised pre-training that uses labelled data. In particular, this pre-training procedure is a simulation that shows the changes in judgment patterns as they progress from primary to mature within the human brain. In this article, the state-of-the-art of neural network pre-training is reviewed. Then, the principles of the auto-encoder and supervised pre-training are introduced in detail. Furthermore, an extended structure of supervised pre-training is proposed. A set of experiments are carried out to compare the performances of different pre-training methods. These experiments include a comparison between the original and pre-trained networks as well as a comparison between the networks with two types of sub-network structures. In addition, a homemade database is established to analyze the influence of pre-training on the generalization ability of neural networks. Finally, an ordinary convolutional neural network is used to verify the applicability of supervised pre-training.
- Y. W. Li and L. Ma. 2016. Big data effective information filtering mining of Internet of Things based on SVM. Control Engineering of China 23, 10 (2016), 1533–1537.Google Scholar
- K. M. Bimal and M. A. Gholam. 2012. Differential epidemic model of virus and worms in computer network. International Journal of Network Security 14, 3 (2012), 149–155.Google Scholar
- M. Volmer, B. G. Wolthers, and H. J. Metting et al. 2020. Artificial neural network predictions of urinary calculus compositions analyzed with infrared spectroscopy. Clinical Chemistry 40, 9 (2020), 1692–1697.Google Scholar
Cross Ref
- S. Zhou, J. Wang, and J. Wang et al. 2017. Point to set similarity based deep feature learning for person re-identification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Honolulu, HI. IEEE Computer Society, 3741–3750.Google Scholar
- L. Arel, D. C. Rose, and T. P. Karnowski. 2010. Deep machine learning: a new frontier in artificial intelligence research. Computational Intelligence Magazine 5, 4 (2010), 13–18. Google Scholar
Digital Library
- D. Erhan, Y. Bengio, A. Courville, et al. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research 11, 3 (2010), 625–660. Google Scholar
Digital Library
- G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504–507.Google Scholar
- G. E. Hinton, S. Osinder, and Y. W. Teh. 2006. A fast learning algorithm for deep belief nets. Neural Computation 18, 7 (2006), 1527–1554. Google Scholar
Digital Library
- D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognitive Science 9, 1 (1985), 147–169.Google Scholar
Cross Ref
- E. H. L. Aarts and J. H. M. Korst. 1987. Boltzmann machines and their applications. In PARLE, Parallel Architectures and Languages Europe, J. W. Bakker, A. J. de Nijman, P. C. Treleaven, (Eds.). Springer, Berlin. 34–50. Google Scholar
Digital Library
- P. Vincent, H. Larochelle, and Y. Bengio. 2008. Extracting and composing robust features with denoising auto-encoders. In Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland. 1096–1103. Google Scholar
Digital Library
- P. Vincent, H. Larochelle, I. Lajoie, et al. 2010. Stacked denoising auto-encoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research 11, 12 (2010), 3371–3408. Google Scholar
Digital Library
- R. Salah, P. Vincent, X. Muller, et al. 2011. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on Machine Learning Bellevue, Washington, USA, June 28-July 2. 833–840. Google Scholar
Digital Library
- J. Masci, U. Meier, and D. Ciresan. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. In Proceedings of the 21st International Conference on Artificial Neural Networks. Vol. 6791. Espoo, Finland, June 14-17, 2011. Springer-Verlag, 52--59. Google Scholar
Digital Library
- T. Schlegl, J. Ofner, and G. Langs. 2014. Unsupervised pre-training across image domains improves lung tissue classification. In Medical Computer Vision (MCV’14): Algorithms for Big Data, Cambridge, MA, USA, September 18, 2014. Springer International Publishing, 82--93.Google Scholar
- H. I. Suk, S. W. Lee, and D. Shen. 2013. Latent feature representation with stacked auto-encoder for AD/MCI diagnosis. Brain Structure and Function 220, 2 (2023), 841–859.Google Scholar
Cross Ref
- P. O. Glauner. 2015. Deep convolutional neural networks for smile recognition. In Deep Learning For Smile Recognition[C]// 12th Conference on Uncertainty Modelling in Knowledge Engineering and Decision Making (FLINS'16), Roubaix, France, August 24-26, 2016. 986--989.Google Scholar
- J. Xu, L. Xiang, Q. Liu, et al. 2016. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Transactions on Medical Imaging 35, 1 (2016), 119.Google Scholar
Cross Ref
- A. Santara, D. Maji, D. Tejas, et al. 2016. Faster learning of deep stacked autoencoders on multi-core systems using synchronized layer-wise pre-training. arXiv preprint arXiv:1603.02836.Google Scholar
- Z. Zhang, Y. Song, and H. Qi. 2017. Age progression/regression by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17), Hawaii, USA July 21-26, 2017. 287--291.Google Scholar
- B. Du, W. Xiong, and J. Wu. 2017. Stacked convolutional denoising auto-encoders for feature representation. IEEE Transactions on Cybernetics 47, 4 (2017), 1017–1027.Google Scholar
Cross Ref
- Y. Bi, P. Wang, X. Guo, et al. 2019. K-means clustering optimizing deep stacked sparse autoencoder. Sensing and Imaging: An International Journal 20, 1 (2019), 6.Google Scholar
Cross Ref
- L. Deng and D. Yu. 2013. Deep learning: Methods and applications. Foundations and Trends in Signal Processing 7, 3.Google Scholar
- Y. Sun, H. Mao, Q. Guo, et al. 2016. Learning a good representation with unsymmetrical auto-encoder. Neural Computing and Applications 27, 5 (2016), 1361–1367. Google Scholar
Digital Library
- B. Schölkopf, J. Platt, and T. Hofmann. 2007. Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19 (2007), 153--160. Google Scholar
Digital Library
- A. Achille and S. Soatto. 2018. Emergence of invariance and disentanglement in deep representations. In IEEE Information Theory and Applications Workshop (ITA’18), San Diego, CA, February 11–16, 2018. arXiv: 1706.01350v2 [cs.LG] 16, 1–9.Google Scholar
- P. Koprinkova-Hristova, V. Mladenov, and K. N. Kasabov. 2015. How to Pretrain Deep Boltzmann Machines in Two Stages[J]. Springer International Publishing. 10.1007/978-3-319-09903-3 (Chapter 10), 201--219.Google Scholar
- M. Eastwood and C. Jayne. 2013. Restricted Boltzmann machines for pre-training deep Gaussian networks. In Proceedings of International Joint Conference on Neural Networks, Dallas, USA, August 4-9, 2013. 1501--1507.Google Scholar
- G. Desjardins, A. Courville, and Y. Bengio. 2012. On training deep Boltzmann machines. arXiv: 1203.4416v1 [cs.NE] 20, 1–7.Google Scholar
- D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations. MIT Press, 318--362. Google Scholar
Digital Library
Index Terms
Research On Pre-Training Method and Generalization Ability of Big Data Recognition Model of the Internet of Things
Recommendations
A fast and efficient pre-training method based on layer-by-layer maximum discrimination for deep neural networks
In this paper, through extension of the present methods and based on error minimization, two fast and efficient layer-by-layer pre-training methods are proposed for initializing deep neural network (DNN) weights. Due to confrontation with a large number ...
Deep Learning Model and Its Application in Big Data
Design, User Experience, and Usability: Theory and PracticeAbstractIn the era of big data, many of the data that previously seemed hard to collect and use began to be utilized, resulting in an increase of millions of the data to be processed. In order to obtain valuable information, large-scale data can be ...
The role of big data analytics in Internet of Things
The explosive growth in the number of devices connected to the Internet of Things (IoT) and the exponential increase in data consumption only reflect how the growth of big data perfectly overlaps with that of IoT. The management of big data in a ...






Comments