Abstract
Meta-learning approaches have recently achieved promising performance in multi-class incremental learning. However, meta-learners still suffer from catastrophic forgetting, i.e., they tend to forget the learned knowledge from the old tasks when they focus on rapidly adapting to the new classes of the current task. To solve this problem, we propose a novel distilled meta-learning (DML) framework for multi-class incremental learning that integrates seamlessly meta-learning with knowledge distillation in each incremental stage. Specifically, during inner-loop training, knowledge distillation is incorporated into the DML to overcome catastrophic forgetting. During outer-loop training, a meta-update rule is designed for the meta-learner to learn across tasks and quickly adapt to new tasks. By virtue of the bilevel optimization, our model is encouraged to reach a balance between the retention of old knowledge and the learning of new knowledge. Experimental results on four benchmark datasets demonstrate the effectiveness of our proposal and show that our method significantly outperforms other state-of-the-art incremental learning methods.
- [1] . 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV’18), , , , and (Eds.). Springer International Publishing, Cham, 144–161. Google Scholar
Digital Library
- [2] . 1990. Learning a Synaptic Learning Rule. Citeseer.Google Scholar
- [3] . 2018. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision. 241–257.Google Scholar
Digital Library
- [4] . 2018. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV’18). 532–547.Google Scholar
Digital Library
- [5] . 2021. Meta-learning-based incremental few-shot object detection. IEEE Trans Circ. Syst. Vid. Technol. 32, 4 (2021), 2158–2169.Google Scholar
Cross Ref
- [6] . 2022. MetaFSCIL: A meta-learning approach for few-shot class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14166–14175.Google Scholar
Cross Ref
- [7] . 2015. Branch-specific dendritic Ca2+ spikes cause persistent synaptic plasticity. Nature 520, 7546 (2015), 180–185.Google Scholar
- [8] . 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems. 6470–6479.Google Scholar
- [9] . 2019. Learning without memorizing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). 5133–5141.
DOI: Google ScholarCross Ref
- [10] . 2015. Learning in nonstationary environments: A survey. IEEE Comput. Intell. Mag. 10, 4 (2015), 12–25.Google Scholar
Digital Library
- [11] . 2021. PLOP: Learning without forgetting for continual semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’21). 4040–4050.Google Scholar
Cross Ref
- [12] . 2015. Distilling the knowledge in a neural network. Comput. Sci. 14, 7 (2015), 38–39.Google Scholar
- [13] . 2019. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). 831–839.
DOI: Google ScholarCross Ref
- [14] . 2019. Meta-learning representations for continual learning. In Advances in Neural Information Processing Systems, Vol. 32. 1–15.Google Scholar
- [15] . 2017. Overcoming catastrophic forgetting in neural networks. Proc. Natl. Acad. Sci. U.S.A. 114, 13 (
March 2017), 3521–3526.DOI: Google ScholarCross Ref
- [16] . 2021. Incremental object detection via meta-learning. IEEE Trans. Pattern Anal. Mach. Intell. (2021), 1–11.Google Scholar
- [17] . 2009. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Technical Report 1 (
01 2009).Google Scholar - [18] . 2002. On the power of incremental learning. Theor. Comput. Sci. 2, 288 (2002), 277–307.Google Scholar
Digital Library
- [19] . 2010. The mnist Database of Handwritten Digits. Retrieved from http://yann.lecun.com/exdb/mnist/.Google Scholar
- [20] . 2017. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell. 40, 12 (2017), 2935–2947.Google Scholar
Digital Library
- [21] . 2019. On the variance of the adaptive learning rate and beyond. arXiv:1908.03265. Retrieved from https://arxiv.org/abs/1908.03265.Google Scholar
- [22] . 2018. Rotate your Networks: Better weight consolidation and less catastrophic forgetting. In Proceedings of the 24th International Conference on Pattern Recognition (ICPR’18). 2262–2268.
DOI: Google ScholarCross Ref
- [23] . 2021. Structural knowledge organization and transfer for class-incremental learning. In Proceedings of the ACM Multimedia Asia (MMAsia’21). Association for Computing Machinery, New York, NY, Article
18 , 7 pages.DOI: Google ScholarDigital Library
- [24] . 2021. Adaptive aggregation networks for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2544–2553.Google Scholar
Cross Ref
- [25] . 2020. Mnemonics Training: Multi-class incremental learning without forgetting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE, 12245–12254.
DOI: Google ScholarCross Ref
- [26] . 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation. Elsevier, 109–165.
DOI: Google ScholarCross Ref
- [27] . 2013. The stability-plasticity dilemma: Investigating the continuum from catastrophic forgetting to age-limited learning effects. Front. Psychol. 4, 504 (2013), 1–3.
DOI: Google ScholarCross Ref
- [28] . 2009. Learn\(^{++}\).NC: Combining ensemble of classifiers with dynamically weighted consult-and-vote for efficient incremental learning of new classes. IEEE Trans. Neural Netw. 20, 1 (2009), 152–168.Google Scholar
Digital Library
- [29] . 2011. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011. 1–9.Google Scholar
- [30] . 2018. On first-order meta-learning algorithms. arXiv:1803.02999. Retrieved from https://arxiv.org/abs/1803.02999.Google Scholar
- [31] . 2018. Reptile: A scalable metalearning algorithm. arXiv:1803.02999. Retrieved from https://arxiv.org/abs/1803.02999.Google Scholar
- [32] . 2019. Random path selection for incremental learning. Advances in Neural Information Processing Systems (2019), 1–11.Google Scholar
- [33] . 2020. iTAML: An incremental task-agnostic meta-learning approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’20). IEEE, 13588–13597.
DOI: Google ScholarCross Ref
- [34] . 2017. iCaRL: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 5533–5542.
DOI: Google ScholarCross Ref
- [35] . 2001. Incremental learning with support vector machines. In Proceedings of the IEEE International Conference on Data Mining. 641–642.
DOI: Google ScholarCross Ref
- [36] . 2015. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115, 3 (2015), 211–252.Google Scholar
Digital Library
- [37] . 1992. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Comput. 4, 1 (1992), 131–139.Google Scholar
Digital Library
- [38] . 2017. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, , , , , , , and (Eds.), Vol. 30. 2290–2999.Google Scholar
- [39] . 2022. Graph few-shot class-incremental learning. In Proceedings of the 15th ACM International Conference on Web Search and Data Mining. 987–996.Google Scholar
Digital Library
- [40] . 2022. Incremental meta-learning via episodic replay distillation for few-shot image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3729–3739.Google Scholar
Cross Ref
- [41] . 2019. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). 374–382.
DOI: Google ScholarCross Ref
- [42] . 2022. RD-IOD: Two-level residual-distillation-based triple-network for incremental object detection. ACM Trans. Multimedia Comput. Commun. Appl. 18, 1 (2022), 1–23.Google Scholar
Digital Library
- [43] . 2017. Continual learning through synaptic intelligence. Int. Conf. Mach. Learn. 70 (2017), 3987–3995.Google Scholar
- [44] . 2020. Class-incremental learning via deep model consolidation. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV’20). IEEE, 1131–1140.
DOI: Google ScholarCross Ref
- [45] . 2021. Co-transport for class-incremental learning. In Proceedings of the 29th ACM International Conference on Multimedia. 1645–1654.Google Scholar
Digital Library
Index Terms
Distilled Meta-learning for Multi-Class Incremental Learning
Recommendations
Incremental learning with neural networks for computer vision: a survey
AbstractIncremental learning is one of the most important abilities of human beings. In the age of artificial intelligence, it is the key task to make neural network models as powerful as human beings, to achieve the ability to continuously acquire, fine-...
Homomorphisms Between Transfer, Multi-task, and Meta-learning Systems
Artificial General IntelligenceAbstractTransfer learning, multi-task learning, and meta-learning are well-studied topics concerned with the generalization of knowledge across learning tasks and are closely related to general intelligence. But, the formal, general systems differences ...
INFGMN Incremental Neuro-Fuzzy Gaussian mixture network
A NFS that learns incrementally using a single scan over the training data.The learning process can proceed in perpetuity as new training data become available.A MamdaniLarsen fuzzy rule base is defined automatically and incrementally.Attempts to ...






Comments