Self-supervised learning by information enhancement: target-generating and spontaneous learning for competitive learning

ABSTRACT

In this paper, we propose a new self-supervised learning method for competitive learning as well as self-organizing maps. In this model, a network enhances its state by itself, and this enhanced state is to be imitated by another state of the network. We set up an enhanced and a relaxed state, and the relaxed state tries to imitate the enhanced state as much as possible by minimizing the free energy. To demonstrate the effectiveness of this method, we apply information enhancement learning to the SOM. For this purpose, we introduce collectiveness, in which all neurons collectively respond to input patterns, into an enhanced state. Then, this enhanced and collective state should be imitated by the other non-enhanced and relaxed state. We applied the method to an artificial data and three data from the well-known machine learning database. Experimental results showed that the U-matrices obtained were significantly similar to those produced by the conventional SOM. However, better performance could be obtained in terms of quantitative and topological errors. The experimental results suggest the possibility for self-supervised learning to be applied to many different neural network models.

References

1. X. Zhu, "Semi-supervised learning literature survey," Tech. Rep. 1530, Compuer sciences, University of Wisconsin-Madison, 2005.
2. O. Z. Chapell and B. Scholkopf, eds., Semi-supervised learning. MIT Press, 2005.
3. D. E. Rumelhart and D. Zipser, "Feature discovery by competitive learning," Cognitive Science, vol. 9, pp. 75-112, 1985.
4. K. Fukushima, "Cognitron: a self-organizing multi-layered neural network," Biological Cybernetics, vol. 20, pp. 121-136, 1975.
5. K. Fukushima, "Neocognitron: a hierarchical neural network capable of visual pattern recognition," Biological Cybernetics, vol. 20, pp. 121-136, 1975.
6. K. Fukushima, "Neocognitron: a neural network model for a mechanism of visual pattern recognition," IEEE Transactions on Systems, Man, and Cyvernetics, vol. 13, pp. 826-834, 1983.
7. T. Kohonen, Self-Organization and Associative Memory. New York: Springer-Verlag, 1988.
8. T. Kohonen, Self-Organizing Maps. Springer-Verlag, 1995.
9. S. C. Ahalt, A. K. Krishnamurthy, P. Chen, and D. E. Melton, "Competitive learning algorithms for vector quantization," Neural Networks, vol. 3, pp. 277-290, 1990.
10. L. Xu, "Rival penalized competitive learning for clustering analysis, RBF net, and curve detection," IEEE Transaction on Neural Networks, vol. 4, no. 4, pp. 636-649, 1993.
11. A. Luk and S. Lien, "Properties of the generalized lotto-type competitive learning," in Proceedings of International conference on neural information processing, (San Mateo: CA), pp. 1180-1185, Morgan Kaufmann Publishers, 2000.
12. M. M. Van Hulle, "Topographic map formation by maximizing unconditional entropy: a plausible strategy for 'on-line' unsupervised competitive learning and nonparametric density estimation," IEEE Transactions on Neural Networks, vol. 7, no. 5, pp. 1299-1305, 1996.
13. M. M. V. Hulle, "The formation of topographic maps that maximize the average mutual information of the output responses to noiseless input signals," Neural Computation, vol. 9, no. 3, pp. 595-606, 1997.
14. M. M. Van Hulle, "Faithful representations with topographic maps," Neural Networks, vol. 12, no. 6, pp. 803-823, 1999.
15. Y. J. Zhang and Z. Q. Liu, "Self-splitting competitive learning: a new on-line clustering paradigm," IEEE Transactions on Neural Networks, vol. 13, no. 2, pp. 369-380, 2002.
16. H. Xiong, M. N. S. Swamy, and M. O. Ahmad, "Competitive splitting for codebook initialization," IEEE Signal Processing Letters, vol. 11, pp. 474-477, 2004.
17. J. C. Yen, J. I. Guo, and H. C. Chen, "A new k-winners-take-all neural networks and its array architecture," IEEE Transactions on Neural Networks, vol. 9, no. 5, pp. 901-912, 1998.
18. S. Ridella, S. Rovetta, and R. Zunino, "K-winner machines for pattern classification," IEEE Transactions on Neural Networks, vol. 12, no. 2, pp. 371-385, 2001.
19. R. Kamimura and T. Kamimura, "Structural information and linguistic rule extraction," in Proceedings of ICONIP-2000, pp. 720-726, 2000.
20. R. Kamimura, T. Kamimura, and O. Uchida, "Flexible feature discovery and structural information control," Connection Science, vol. 13, no. 4, pp. 323-347, 2001.
21. R. Kamimura, "Information-theoretic competitive learning with inverse euclidean distance," Neural Processing Letters, vol. 18, pp. 163-184, 2003.
22. R. Kamimura, "Teacher-directed learning: information-theoretic competitive learning in supervised multi-layered networks," Connection Science, vol. 15, pp. 117-140, 2003.
23. R. Kamimura, "Progressive feature extraction by greedy network-growing algorithm," Complex Systems, vol. 14, no. 2, pp. 127-153, 2003.
24. R. Kamimura, "Information theoretic competitive learning in self-adaptive multi-layered networks," Connection Science, vol. 13, no. 4, pp. 323-347, 2003.
25. K. Rose, E. Gurewitz, and G. C. Fox, "Statistical mechanics and phase transition in clustering," Physical review letters, vol. 65, no. 8, pp. 945- 948, 1990.
26. K. Rose, E. Gurewitz, and G. C. Fox, "Vector quantization by deterministic annealing," IEEE transactions on information theory, vol. 38, no. 4, pp. 1249-1257, 1992.
27. N. Ueda and R. Nakano, "Deterministic annealing variant of the em algorithm," in Advances in Neural Information Processing Systems, pp. 545-552, 1995.
28. N. Ueda and R. Nakano, "Deterministic annealing em algorithm," Neural Networks, vol. 11, pp. 271-282, 1998.
29. C. M. Bishop, Neural networks for pattern recognition. Oxford University Press, 1995.
30. T. Graepel, M. Burger, and K. Obermayer, "Phase transitions in stochastic self-organizing maps," Physical Review E, vol. 56, no. 4, p. 3876, 1997.
31. T. Heskes, "Self-organizing maps, vector quantization, and mixture modeling," IEEE Transactions on Neural Networks, vol. 12, no. 6, pp. 1299-1305, 2001.
32. R. Kamimura, "Free energy-based competitive learning for mutual information maximization," in Proceedings of IEEE Conference on Systems, Man, and Cybernetics, pp. 223-227, 2008.
33. R. Kamimura, "Free energy-based competitive learning for self-organizing maps," in Proceedings of Artificial Intelligence and Applications , pp. 414-419, 2008.
34. T. Kohonen, "The self-organizing maps," Proceedings of the IEEE, vol. 78, no. 9, pp. 1464-1480, 1990.
35. G. J. Goodhill and T. J. Sejnowski, "A unifying objective function for topographic mappings," Neural Computation, vol. 9, pp. 1291-1303, 1997.
36. S. P. Luttrell, "A Bayesian analysis of self-organising maps," Neural Computation, vol. 6, no. 5, pp. 767-794, 1994.
37. B. Bakker and T. Heskes, "Clustering ensembles of neural network models," Neural Networks, vol. 16, pp. 261-269, 2003.
38. A. Utsugi, "Hyperparameter selection for self-organizing maps," Neural Computation, vol. 9, no. 3, pp. 623-635, 1997.
39. A. Utsugi, "Density estimation by mixture models with smoothing priors," Neural Computation, vol. 10, pp. 2115-2135, 1998.
40. R. Linsker, "Self-organization in a perceptual network," Computer, vol. 21, pp. 105-117, 1988.
41. R. Linsker, "How to generate ordered maps by maximizing the mutual information between input and output," Neural Computation, vol. 1, pp. 402-411, 1989.
42. R. Linsker, "Local synaptic rules suffice to maximize mutual information in a linear network," Neural Computation, vol. 4, pp. 691-702, 1992.
43. R. Linsker, "Improved local learning rule for information maximization and related applications," Neural Networks, vol. 18, pp. 261-265, 2005.
44. R. Kamimura, "Cooperative information maximization with gauissian activation functions for self-organizing maps," IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 909-919, 2006.
45. T. M. Martinez, S. G. Berkovich, and K. J. Schulten, "Neural-gas network for vector quanitization and its application to time-series prediction," IEEE transactions on neural networks, vol. 4, no. 4, pp. 558-569, 1993.
46. D. Erdogmus and J. Principe, "Lower and upper bounds for misclassification probability based on renyi's information," Journal of VLSI signal processing systems, vol. 37, no. 2/3, pp. 305-317, 2004.
47. K. Torkkola, "Feature extraction by non-parametric mutual information maximization," Journal of Machine Learning Research, vol. 3, pp. 1415- 1438, 2003.
48. J. Versanto, J. Himberg, E. ALhoniemi, and J. Parhankagas, "Som toolbox for matlab 5," Tech. Rep. A57, Helsinki University of Technology, 2000.
49. S. Kaski, J. Nikkila, and T. Kohonen, "Methods for interpreting a self-organized map in data analysis," in Proceedings of ESANN'98, 6th European Symposium on Artificial Neural Networks, Bruges, April 22-24 (M. Verleysen, ed.), pp. 185-190, Brussels, Belgium: D-Facto, 1998.
50. J. Vesanto, "SOM-based data visualization methods," Intelligent-Data-Analysis , vol. 3, pp. 111-26, 1999.
51. J. Vesanto and E. Alhoniemi, "Clustering of the self-organizing map," IEEE-NN, vol. 11, p. 586, May 2000.

Index Terms

• Computing methodologies

  • Machine learning

    • Machine learning approaches

      • Neural networks