skip to main content
research-article

Research On Pre-Training Method and Generalization Ability of Big Data Recognition Model of the Internet of Things

Authors Info & Claims
Published:21 July 2021Publication History
Skip Abstract Section

Abstract

The Internet of Things and big data are currently hot concepts and research fields. The mining, classification, and recognition of big data in the Internet of Things system are the key links that are widely of concern at present. The artificial neural network is beneficial for multi-dimensional data classification and recognition because of its strong feature extraction and self-learning ability. Pre-training is an effective method to address the gradient diffusion problem in deep neural networks and could result in better generalization. This article focuses on the performance of supervised pre-training that uses labelled data. In particular, this pre-training procedure is a simulation that shows the changes in judgment patterns as they progress from primary to mature within the human brain. In this article, the state-of-the-art of neural network pre-training is reviewed. Then, the principles of the auto-encoder and supervised pre-training are introduced in detail. Furthermore, an extended structure of supervised pre-training is proposed. A set of experiments are carried out to compare the performances of different pre-training methods. These experiments include a comparison between the original and pre-trained networks as well as a comparison between the networks with two types of sub-network structures. In addition, a homemade database is established to analyze the influence of pre-training on the generalization ability of neural networks. Finally, an ordinary convolutional neural network is used to verify the applicability of supervised pre-training.

References

  1. Y. W. Li and L. Ma. 2016. Big data effective information filtering mining of Internet of Things based on SVM. Control Engineering of China 23, 10 (2016), 1533–1537.Google ScholarGoogle Scholar
  2. K. M. Bimal and M. A. Gholam. 2012. Differential epidemic model of virus and worms in computer network. International Journal of Network Security 14, 3 (2012), 149–155.Google ScholarGoogle Scholar
  3. M. Volmer, B. G. Wolthers, and H. J. Metting et al. 2020. Artificial neural network predictions of urinary calculus compositions analyzed with infrared spectroscopy. Clinical Chemistry 40, 9 (2020), 1692–1697.Google ScholarGoogle ScholarCross RefCross Ref
  4. S. Zhou, J. Wang, and J. Wang et al. 2017. Point to set similarity based deep feature learning for person re-identification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17), Honolulu, HI. IEEE Computer Society, 3741–3750.Google ScholarGoogle Scholar
  5. L. Arel, D. C. Rose, and T. P. Karnowski. 2010. Deep machine learning: a new frontier in artificial intelligence research. Computational Intelligence Magazine 5, 4 (2010), 13–18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. D. Erhan, Y. Bengio, A. Courville, et al. 2010. Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research 11, 3 (2010), 625–660. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. Science 313, 5786 (2006), 504–507.Google ScholarGoogle Scholar
  8. G. E. Hinton, S. Osinder, and Y. W. Teh. 2006. A fast learning algorithm for deep belief nets. Neural Computation 18, 7 (2006), 1527–1554. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. D. H. Ackley, G. E. Hinton, and T. J. Sejnowski. 1985. A learning algorithm for Boltzmann machines. Cognitive Science 9, 1 (1985), 147–169.Google ScholarGoogle ScholarCross RefCross Ref
  10. E. H. L. Aarts and J. H. M. Korst. 1987. Boltzmann machines and their applications. In PARLE, Parallel Architectures and Languages Europe, J. W. Bakker, A. J. de Nijman, P. C. Treleaven, (Eds.). Springer, Berlin. 34–50. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. P. Vincent, H. Larochelle, and Y. Bengio. 2008. Extracting and composing robust features with denoising auto-encoders. In Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland. 1096–1103. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. P. Vincent, H. Larochelle, I. Lajoie, et al. 2010. Stacked denoising auto-encoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research 11, 12 (2010), 3371–3408. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. R. Salah, P. Vincent, X. Muller, et al. 2011. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on Machine Learning Bellevue, Washington, USA, June 28-July 2. 833–840. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. J. Masci, U. Meier, and D. Ciresan. 2011. Stacked convolutional auto-encoders for hierarchical feature extraction. In Proceedings of the 21st International Conference on Artificial Neural Networks. Vol. 6791. Espoo, Finland, June 14-17, 2011. Springer-Verlag, 52--59. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. T. Schlegl, J. Ofner, and G. Langs. 2014. Unsupervised pre-training across image domains improves lung tissue classification. In Medical Computer Vision (MCV’14): Algorithms for Big Data, Cambridge, MA, USA, September 18, 2014. Springer International Publishing, 82--93.Google ScholarGoogle Scholar
  16. H. I. Suk, S. W. Lee, and D. Shen. 2013. Latent feature representation with stacked auto-encoder for AD/MCI diagnosis. Brain Structure and Function 220, 2 (2023), 841–859.Google ScholarGoogle ScholarCross RefCross Ref
  17. P. O. Glauner. 2015. Deep convolutional neural networks for smile recognition. In Deep Learning For Smile Recognition[C]// 12th Conference on Uncertainty Modelling in Knowledge Engineering and Decision Making (FLINS'16), Roubaix, France, August 24-26, 2016. 986--989.Google ScholarGoogle Scholar
  18. J. Xu, L. Xiang, Q. Liu, et al. 2016. Stacked sparse autoencoder (SSAE) for nuclei detection on breast cancer histopathology images. IEEE Transactions on Medical Imaging 35, 1 (2016), 119.Google ScholarGoogle ScholarCross RefCross Ref
  19. A. Santara, D. Maji, D. Tejas, et al. 2016. Faster learning of deep stacked autoencoders on multi-core systems using synchronized layer-wise pre-training. arXiv preprint arXiv:1603.02836.Google ScholarGoogle Scholar
  20. Z. Zhang, Y. Song, and H. Qi. 2017. Age progression/regression by conditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR'17), Hawaii, USA July 21-26, 2017. 287--291.Google ScholarGoogle Scholar
  21. B. Du, W. Xiong, and J. Wu. 2017. Stacked convolutional denoising auto-encoders for feature representation. IEEE Transactions on Cybernetics 47, 4 (2017), 1017–1027.Google ScholarGoogle ScholarCross RefCross Ref
  22. Y. Bi, P. Wang, X. Guo, et al. 2019. K-means clustering optimizing deep stacked sparse autoencoder. Sensing and Imaging: An International Journal 20, 1 (2019), 6.Google ScholarGoogle ScholarCross RefCross Ref
  23. L. Deng and D. Yu. 2013. Deep learning: Methods and applications. Foundations and Trends in Signal Processing 7, 3.Google ScholarGoogle Scholar
  24. Y. Sun, H. Mao, Q. Guo, et al. 2016. Learning a good representation with unsymmetrical auto-encoder. Neural Computing and Applications 27, 5 (2016), 1361–1367. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. B. Schölkopf, J. Platt, and T. Hofmann. 2007. Greedy layer-wise training of deep networks. Advances in Neural Information Processing Systems 19 (2007), 153--160. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. A. Achille and S. Soatto. 2018. Emergence of invariance and disentanglement in deep representations. In IEEE Information Theory and Applications Workshop (ITA’18), San Diego, CA, February 11–16, 2018. arXiv: 1706.01350v2 [cs.LG] 16, 1–9.Google ScholarGoogle Scholar
  27. P. Koprinkova-Hristova, V. Mladenov, and K. N. Kasabov. 2015. How to Pretrain Deep Boltzmann Machines in Two Stages[J]. Springer International Publishing. 10.1007/978-3-319-09903-3 (Chapter 10), 201--219.Google ScholarGoogle Scholar
  28. M. Eastwood and C. Jayne. 2013. Restricted Boltzmann machines for pre-training deep Gaussian networks. In Proceedings of International Joint Conference on Neural Networks, Dallas, USA, August 4-9, 2013. 1501--1507.Google ScholarGoogle Scholar
  29. G. Desjardins, A. Courville, and Y. Bengio. 2012. On training deep Boltzmann machines. arXiv: 1203.4416v1 [cs.NE] 20, 1–7.Google ScholarGoogle Scholar
  30. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning internal representations by error propagation. In Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations. MIT Press, 318--362. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Research On Pre-Training Method and Generalization Ability of Big Data Recognition Model of the Internet of Things

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    • Published in

      cover image ACM Transactions on Asian and Low-Resource Language Information Processing
      ACM Transactions on Asian and Low-Resource Language Information Processing  Volume 20, Issue 5
      September 2021
      320 pages
      ISSN:2375-4699
      EISSN:2375-4702
      DOI:10.1145/3467024
      Issue’s Table of Contents

      Copyright © 2021 Association for Computing Machinery.

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 21 July 2021
      • Accepted: 1 November 2020
      • Revised: 1 October 2020
      • Received: 1 April 2020
      Published in tallip Volume 20, Issue 5

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article
      • Refereed

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format .

    View HTML Format
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!