Multi-Label Active Learning Algorithms for Image Classification: Overview and Future Promise

Abstract

Image classification is a key task in image understanding, and multi-label image classification has become a popular topic in recent years. However, the success of multi-label image classification is closely related to the way of constructing a training set. As active learning aims to construct an effective training set through iteratively selecting the most informative examples to query labels from annotators, it was introduced into multi-label image classification. Accordingly, multi-label active learning is becoming an important research direction. In this work, we first review existing multi-label active learning algorithms for image classification. These algorithms can be categorized into two top groups from two aspects respectively: sampling and annotation. The most important component of multi-label active learning is to design an effective sampling strategy that actively selects the examples with the highest informativeness from an unlabeled data pool, according to various information measures. Thus, different informativeness measures are emphasized in this survey. Furthermore, this work also makes a deep investigation on existing challenging issues and future promises in multi-label active learning with a focus on four core aspects: example dimension, label dimension, annotation, and application extension.

References

1. Rakesh Agrawal, Tomasz Imielinski, and Arun Swami. 1993. Mining association rules between sets of items in large databases. In Proceedings of the ACM International Conference on Management of Data (SIGMOD’93). 207--216.
2. Erin L. Allwein, Robert E. Schapire, and Yoram Singer. 2000. Reducing multiclass to binary: A unifying approach for margin classifiers. In Proceedings of the 17th International Conference on Machine Learning (ICML’00). 9--16.
3. Massih R. Amini and Patrick Gallinari. 2005. Semi-supervised learning with an imperfect supervisor. Knowl. Inf. Syst. 13, 1 (2005), 1--42.
4. Dana Angluin. 1988. Queries and concept learning. Mach. Learn. 2, 4 (1988), 319--342.
5. Mustafa Bilgic. 2012. Combining active learning and dynamic dimensionality reduction. In Proceedings of the 2012 SIAM International Conference on Data Mining. SIAM, 696--707.
6. Hans Kristian Bø, Ole Solheim, Asgeir Store Jakola, Kjell-Arne Kvistad, Ingerid Reinertsen, and Erik Magnus Berntsen. 2017. Intra-rater variability in low-grade glioma segmentation. J. Neuro-oncol. 131, 2 (2017), 393--402.
7. Matthew R. Boutell, Jiebo Luo, Xipeng Shen, and Christopher M. Brown. 2004. Learning multi-label scene classification. Pattern Recogn. 37, 9 (2004), 1757--1771.
8. Selcuk Bucak, Serhat, Rong Jin, and Anil K. Jain. 2011. Multi-label learning with incomplete class assignments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’11). 2801--2808.
9. Xiangyong Cao, Yang Chen, Qian Zhao, Deyu Meng, Yao Wang, Dong Wang, and Zongben Xu. 2016. Low-rank matrix factorization under general mixture noise distributions. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’16). 1493--1501.
10. Shayok Chakraborty, Vineeth Balasubramanian, and Sethuraman Panchanathan. 2011. Optimal batch selection for active learning in multi-label classification. In Proceedings of the 19th ACM International Conference on Multimedia. ACM, 1413--1416.
11. Beijing Chen, Huazhong Shu, Gouenou Coatrieux, Gang Chen, Xingming Sun, and Jean Louis Coatrieux. 2015. Color image analysis by quaternion-type moments. J. Math. Imag. Vis. 51, 1 (2015), 124--144.
12. Gang Chen, Yangqiu Song, Fei Wang, and Changshui Zhang. 2008. Semi-supervised multi-label learning by solving a sylvester equation. In Proceedings of the SIAM International Conference on Data Mining (SDM’08). 410--419.
13. Myung Jin Choi, Antonio Torralba, and Alan S. Willsky. 2012. A tree-based context model for object recognition. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2 (2012), 240--252.
14. David Cohn, Les Atlas, and Richard Ladner. 1994. Improving generalization with active learning. Mach. Learn. 15, 2 (1994), 201--221.
15. David. A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1996. Active learning with statistical models. J. Artif. Intell. Res. 4, 1 (1996), 705--712.
16. Ido Dagan and Sean P. Engelson. 1995. Committee-based sampling for training probabilistic classifiers. In Proceedings of the 12th International Conference on Machine Learning (ICML’95). 150--157.
17. Pinar Donmez, Jaime G. Carbonell, and Jeff Schneider. 2009. Efficiently learning the accuracy of labeling sources for selective sampling. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’09). 259--268.
18. Charles Elkan and Keith Noto. 2008. Learning classifiers from only positive and unlabeled data. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’08). 213--220.
19. Jonathan J. Entis, Priya Doerga, Lisa Feldman Barrett, and Bradford C. Dickerson. 2012. A reliable protocol for the manual segmentation of the human amygdala and its subregions using ultra-high resolution MRI. Neuroimage 60, 2 (2012), 1226--1235.
20. Andrea Esuli and Fabrizio Sebastiani. 2009. Active learning strategies for multi-label text classification. In Proceedings of the European Conference on Information Retrieval (ECIR’09). 102--113.
21. Yifan Fu, Xingquan Zhu, and Bin Li. 2013. A survey on instance selection for active learning. Knowl. Inf. Syst. 35, 2 (2013), 249--283.
22. Atsushi Fujii, Takenobu Tokunaga, Kentaro Inui, and Hozumi Tanaka. 1998. Selective sampling for example-based word sense disambiguation. Comput. Linguist. 24, 4 (1998), 573--597.
23. Gabriel Pui Cheong Fung, Jeffrey X. Yu, Hongjun Lu, and Philip S. Yu. 2006. Text classification without negative examples revisit. IEEE Trans. Knowl. Data Eng. 18, 1 (2006), 6--20.
24. Nengneng Gao, Sheng-Jun Huang, and Songcan Chen. 2016. Multi-label active learning by model guided distribution matching. Front. Comput. Sci. 10, 5 (2016), 845--855.
25. Eva Gibaja and Sebastián Ventura. 2015. A tutorial on multilabel learning. ACM Comput. Surv. 47, 3 (2015), 52.
26. Robert J. Gillies, Paul E. Kinahan, and Hedvig Hricak. 2015. Radiomics: Images are more than pictures, they are data. Radiology 278, 2 (2015), 563--577.
27. Bin Gu and Victor S. Sheng. 2016. A robust regularization path algorithm for v-support vector classification. IEEE Trans. Neur. Netw. Learn. Syst. 1, 99 (2016), 1--8.
28. Bin Gu, Victor S. Sheng, and Shuo Li. 2015. Bi-parameter space partition for cost-sensitive SVM. In Proceedings of the International Conference on Artificial Intelligence (AAAI’15). 3532--3539.
29. Bin Gu, Xingming Sun, and Victor. S. Sheng. 2017. Structural minimax probability machine. IEEE Trans. Neur. Netw. Learn. Syst. 28, 7 (2017), 1646--1656.
30. Anqian Guo, Jian Wu, Victor S. Sheng, Pengpeng Zhao, and Zhiming Cui. 2017. Multi-label active learning with low-rank mapping for image classification. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’17). 259--264.
31. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA data mining software: An update. ACM SIGKDD Explor. Newslett. 11, 1 (2009), 10--18.
32. Jeff Howe. 2006. The rise of crowdsourcing. Wired Mag. 14, 6 (2006), 1--4.
33. Xian Sheng Hua and Guo Jun Qi. 2008. Online multi-label active annotation: Towards large-scale content-based video search. In Proceedings of the ACM International Conference on Multimedia (ACM MM’08). 141--150.
34. Shengjun Huang, Rong Jin, and Zhihua Zhou. 2010. Active learning by querying informative and representative examples. In Proceedings of the International Conference on Neural Information Processing Systems. 892--900.
35. Shengjun Huang, Rong Jin, and Zhihua Zhou. 2014. Active learning by querying informative and representative examples. IEEE Trans. Pattern Anal. Mach. Intell. 36, 10 (2014), 1936--49.
36. Sheng Jun Huang, Songcan Chen, and Zhihua Zhou. 2015. Multi-label active learning: Query type matters. International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI’15), 946--952.
37. Yu Gang Jiang, Qi Dai, Jun Wang, Chong Wah Ngo, Xiangyang Xue, and Shih Fu Chang. 2012. Fast semantic diffusion for large-scale context-based image and video annotation. IEEE Trans. Image Process. 21, 6 (2012), 3080--3091.
38. Yang Jiao. 2015. Active Learning-Based Multi-Label Image Classification. Soochow University in China.
39. Yang Jiao, Pengpeng Zhao, Jian Wu, Yujie Shi, and Zhiming Cui. 2014. A multicriterion query-based batch mode active learning technique. In Foundations of Intelligent Systems. 669--680.
40. Yang Jiao, Pengpeng Zhao, Jian Wu, Xuefeng Xian, Haihui Xu, and Zhiming Cui. 2014. Active multi-label learning with optimal label subset selection. In Proceedings of the International Conference on Advanced Data Mining and Applications (ADMA’14). 523--534.
41. Ajay J. Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. 2009. Multi-class active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’09). 2372--2379.
42. Ashish Kapoor, Kristen Grauman, Raquel Urtasun, and Trevor Darrell. 2007. Active learning with gaussian processes for object categorization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’07), Vol. 88. 1--8.
43. Aram Kawewong, Rapeeporn Pimpup, and Osamu Hasegawa. 2013. Incremental learning framework for indoor scene recognition. In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI’13). 496--502.
44. Freda Kemp. 2003. Applied multiple regression/correlation analysis for the behavioral sciences. J. Roy. Stat. Soc. 52, 4 (2003), 691--691.
45. Ross D. King, Jem Rowland, Stephen G. Oliver, Michael Young, Wayne Aubrey, Emma Byrne, Maria Liakata, Magdalena Markham, Pinar Pir, and Larisa N. Soldatova. 2009. The automation of science. Science 324, 5923 (2009), 85--89.
46. Ross D. King, Kenneth E. Whelan, Ffion M. Jones, Philip G. K. Reiser, Christopher H. Bryant, Stephen H. Muggleton, Douglas B. Kell, and Stephen G. Oliver. 2004. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature 427, 6971 (2004), 247--252.
47. V. Krishnamurthy. 2002. Algorithms for optimal scheduling and management of hidden Markov model sensors. IEEE Trans. Sign. Process. 50, 6 (2002), 1382--1397.
48. Kenneth Lang and Eric. B. Baum. 1992. Query learning can work poorly when a human oracle is used. In Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN’92).
49. Guang He Lee, Shao Wen Yang, and Shou De Lin. 2016. Toward implicit sample noise modeling: Deviation-driven matrix factorization. arXiv preprint arXiv:1610.09274 (2016).
50. David D. Lewis and Jason Catlett. 1994. Heterogenous uncertainty sampling for supervised learning. In Proceedings of the International Conference on Machine Learning (ICML’94). 148--156.
51. Jian Li, Xiaolong Li, Bin Yang, and Xingming Sun. 2015. Segmentation-based image copy-move forgery detection scheme. IEEE Trans. Inf. Forens. Secur. 10, 3 (2015), 507--518.
52. Xin Li and Yuhong Guo. 2013. Active learning with multi-label SVM classification. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’13). 1479--1485.
53. Xiaoli Li and Bing Liu. 2003. Learning to classify texts using positive and unlabeled data. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’03). 587--592.
54. Xuchun Li, Lei. Wang, and Eric. Sung. 2004. Multilabel SVM active learning for image classification. In Proceedings of the International Conference on Image Processing (ICIP’04), Vol. 4. 2207--2210.
55. Geert Litjens, Thijs Kooi, Babak Ehteshami Bejnordi, Arnaud Arindra Adiyoso Setio, Francesco Ciompi, Mohsen Ghafoorian, Jeroen Awm Van Der Laak, Bram Van Ginneken, and Clara I. Sánchez. 2017. A survey on deep learning in medical image analysis. Med. Image Anal. 42 (2017), 60--88.
56. B. Liu, Y. Dai, X. Li, and W. S. Lee. 2003. Building text classifiers using positive and unlabeled examples. In Proceedings of the IEEE International Conference on Data Mining (ICDM’03). 179--186.
57. Guangcan Liu, Z. Lin, S. Yan, J. Sun, Y. Yu, and Y. Ma. 2013. Robust recovery of subspace structures by low-rank representation. IEEE Trans. Pattern Anal. Mach. Intell. 35, 1 (2013), 171--184.
58. Deyu Meng and Fernando De La Torre. 2013. Robust matrix factorization with unknown noise. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’13), 1337--1344.
59. Kaushik Mitra, Sameer Sheorey, and Rama Chellappa. 2010. Large-scale matrix factorization with missing data under additional constraints. Ophthal. Res. 40, 1 (2010), 35.
60. Gulisong Nasierding and Abbas Z. Kouzani. 2010. Empirical study of multi-label classification methods for image annotation and retrieval. In Proceedings of the International Conference on Digital Image Computing: Techniques and Applications (DICTA’10). 617--622.
61. Reports of International Data Corporation. 2010. Retrieved from http://www.idc.com/prodserv/prodserv.jsp.
62. Fredrik Olsson. 2009. A Literature Survey of Active Machine Learning in the Context of Natural Language Processing. Swedish Institute of Computer Science. Technical Report T2009:06.
63. Guojun Qi, Xiansheng Hua, Yong Rui, Jinhui Tang, and Hongjiang Zhang. 2008. Two-dimensional active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). 1--8.
64. Guojun Qi, Xiansheng Hua, Yong Rui, Jinhui Tang, and Hongjiang Zhang. 2009. Two-dimensional multilabel active learning with an efficient online adaptation model for image classification. IEEE Trans. Pattern Anal. Mach. Intell. 31, 10 (2009), 1880--1897.
65. Oscar Reyes, Abdulrahman H. Altalhi, and Sebastián Ventura. 2018. Statistical comparisons of active learning strategies over multiple datasets. Knowl.-Based Syst. 145 (2018), 274--288.
66. Oscar Reyes, Carlos Morell, and Sebastián Ventura. 2018. Effective active learning strategy for multi-label learning. Neurocomputing 273 (2018), 494--508.
67. Oscar Reyes, Eduardo Pérez, María Del Carmen Rodríguez-Hernández, Habib M. Fardoun, and Sebastián Ventura. 2016. JCLAL: A Java framework for active learning. J. Mach. Learn. Res. 17, 1 (2016), 3271--3275.
68. Oscar Reyes and Sebastián Ventura. 2018. Evolutionary strategy to perform batch-mode active learning on multi-label data. ACM Trans. Intell. Syst. Technol. 9, 4 (2018), 46.
69. Burr Settles. 2010. Active Learning Literature Survey. Computer Science Technical Report 1648, University of Wisconsin-Madison.
70. H. Sebastian Seung, Manfred Opper, and Haim Sompolinsky. 1992. Query by committee. In Proceedings of the The Annual Workshop on Computational Learning Theory (COLT’92). 287--294.
71. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’08). 614--622.
72. Victor S. Sheng and Jing Zhang. 2019. Machine learning with crowdsourcing: A brief summary of the past research and future directions. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’19), Vol. 33. 9837--9843.
73. Mohan Singh and Eoin Curran. 2008. Active learning for multi-label image annotation. In Proceedings of the Irish Conference on Artificial Intelligence and Cognitive Science (AICS’08). 173--182.
74. Asim Smailagic, Pedro Costa, Hae Young Noh, Devesh Walawalkar, Kartik Khandelwal, Adrian Galdran, Mostafa Mirshekari, Jonathon Fagert, Susu Xu, Pei Zhang, et al. 2018. MedAL: Accurate and robust deep active learning for medical image analysis. In Proceedings of the 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA’18). IEEE, 481--488.
75. Maja Stikic, Kristof Van Laerhoven, and Bernt Schiele. 2008. Exploring semi-supervised and active learning for activity recognition. In Proceedings of the IEEE International Symposium on Wearable Computers. 81--88.
76. Michael D. Story and Marco Durante. 2018. Radiogenomics. Medical Physics 45, 11 (2018), e1111--e1122.
77. Yu Yin Sun, Yin Zhang, and Zhi Hua Zhou. 2010. Multi-label learning with weak label. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI’10). 593--598.
78. Jinhui Tang, Zhengun Zha, Dacheng Tao, and Tatseng Chua. 2012. Semantic-gap-oriented active learning for multilabel image annotation. IEEE Trans. Image Process. 21, 4 (2012), 2354--2360.
79. Simon Tong. 2001. Active Learning: Theory and Applications. Stanford University.
80. Simon Tong and Daphne Koller. 2001. Support vector machine active learning with applications to text classification. J. Mach. Learn. Res. 2, 11 (2001), 45--66.
81. Grigorios Tsoumakas, Ioannis Katakis, and David Taniar. 2007. Multi-label classification: An overview. Int. J. Data Warehous. Min. 3, 3 (2007), 1--13.
82. Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2008. Effective and efficient multilabel classification in domains with large number of labels. In Proceedings of the ECML/PKDD Workshop on Mining Multidimensional Data (MMD’08). 30--44.
83. Grigorios Tsoumakas, Ioannis Katakis, and Ioannis Vlahavas. 2009. Mining multi-label data. In Data Mining and Knowledge Discovery Handbook. Springer, 667--685.
84. Grigorios Tsoumakas, Eleftherios Spyromitros-Xioufis, Jozef Vilcek, and Ioannis Vlahavas. 2011. MULAN: A Java library for multi-label learning. J. Mach. Learn. Res. 12, 7 (2011), 2411--2414.
85. Devis Tuia, Michele Volpi, Loris Copa, Mikhail Kanevski, and Jordi Munoz-Mari. 2011. A survey of active learning algorithms for supervised remote sensing image classification. IEEE J. Select. Top. Sign. Process. 5, 3 (2011), 606--617.
86. Gokhan Tur, Dilek Hakkani-Tur, and Robert E. Schapire. 2005. Combining active and semi-supervised learning for spoken language understanding. Speech Commun. 45, 2 (2005), 171--186.
87. M. Visser, D. M. J. Müller, R. J. M. van Duijn, M. Smits, N. Verburg, E. J. Hendriks, R. J. A. Nabuurs, J. C. J. Bot, R. S. Eijgelaar, M. Witte, et al. 2019. Inter-rater agreement in glioma segmentations on longitudinal MRI. NeuroImage: Clin. 22 (2019), 101727.
88. Meng Wang and Xiansheng Hua. 2011. Active learning in multimedia annotation and retrieval: A survey. ACM Trans. Intell. Syst. Technol. 2, 2 (2011), 1--21.
89. Zhihua Wei, Hanli Wang, and Rui Zhao. 2013. Semi-supervised multi-label image classification based on nearest neighbor editing. Neurocomputing 119 (2013), 462--468.
90. Xuezhi Wen, Ling Shao, Yu Xue, and Wei Fang. 2015. A rapid learning algorithm for vehicle classification. Inf. Sci. 295, 1 (2015), 395--406.
91. Jian Wu, Anqian Guo, Victor S. Sheng, Pengpeng Zhao, Zhiming Cui, and Hua Li. 2017. Adaptive low-rank multi-label active learning for image classification. In Proceedings of the ACM International Conference on Multimedia (ACM MM’17). 1336--1344.
92. Jian Wu, Chunfeng Lian, Su Ruan, Thomas R. Mazur, Sasa Mutic, Mark A. Anastasio, Perry W. Grigsby, Pierre Vera, and Hua Li. 2019. Treatment outcome prediction for cancer patients based on radiomics and belief function theory. IEEE Trans. Radiat. Plasma Med. Sci. 3, 2 (2019), 216--224.
93. Jian Wu, Su Ruan, Chunfeng Lian, Sasa Mutic, Mark A. Anastasio, and Hua Li. 2018. Active learning with noise modeling for medical image annotation. In Proceedings of the 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI’18). IEEE, 298--301.
94. Jian Wu, Victor S. Sheng, Jing Zhang, Pengpeng Zhao, and Zhiming Cui. 2014. Multi-label active learning for image classification. In Proceedings of the IEEE International Conference on Image Processing (ICIP’14). 5227--5231.
95. Jian Wu, Chen Ye, Victor S. Sheng, Yufeng Yao, Pengpeng Zhao, and Zhiming Cui. 2015. Semi-automatic labeling with active learning for multi-label image classification. In Proceedings of the Pacific Rim Conference on Multimedia (PCM’15). 473--482.
96. Jian Wu, Chen Ye, Victor S. Sheng, Jing Zhang, Peng Peng Zhao, and Zhiming Cui. 2017. Active learning with label correlation exploration for multi-label image classification. IET Comput. Vis. 11, 7 (2017), 577--584.
97. Jian Wu, Shiquan Zhao, Victor S. Sheng, Jing Zhang, Chen Ye, Peng Peng Zhao, and Zhiming Cui. 2017. Weak labeled active learning with conditional label dependence for multi-label image classification. IEEE Trans. Multimedia 19, 6 (2017), 1156--1169.
98. Jian Wu, Shiquan Zhao, Victor S. Sheng, Pengpeng Zhao, and Zhiming Cui. 2016. Multi-label active learning for image classification with asymmetrical conditional dependence. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’16). 5227--5231.
99. Zhihua Xia, Xinhui Wang, Xingming Sun, Quansheng Liu, and Naixue Xiong. 2016. Steganalysis of LSB matching using differences between nonadjacent pixels. Multimedia Tools Appl. 75, 4 (2016), 1947--1962.
100. Xin Shun Xu, Yuan Jiang, Liang Peng, Xiangyang Xue, and Zhi Hua Zhou. 2011. Ensemble approach based on conditional random field for multi-label image and video annotation. In Proceedings of the ACM International Conference on Multimedia (ACM MM’11). 1377--1380.
101. Shuicheng Yan. 2012. Practical low-rank matrix approximation under robust L1-norm. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’12). 1410--1417.
102. Bishan Yang, Jian Tao Sun, Tengjiao Wang, and Zheng Chen. 2009. Effective multi-label active learning for text classification. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’09). 917--926.
103. Lin Yang, Yizhe Zhang, Jianxu Chen, Siyuan Zhang, and Danny Z. Chen. 2017. Suggestive annotation: A deep active learning framework for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI’17). Springer, 399--407.
104. Shu Jun Yang, Yuan Jiang, and Zhi Hua Zhou. 2013. Multi-instance multi-label learning with weak label. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI’13). 1862--1868.
105. Chen Ye, Jian Wu, Victor S. Sheng, and Pengpeng Zhao. 2015. Multi-label active learning with label correlation for image classification. In Proceedings of the IEEE International Conference on Image Processing (ICIP’15). 3437--3441.
106. Chen Ye, Jian Wu, Victor S. Sheng, Shiquan Zhao, Pengpeng Zhao, and Zhiming Cui. 2015. Multi-label active learning with chi-square statistics for image classification. In Proceedings of the ACM International Conference on Multimedia Retrieval (ICMR’15). 583--586.
107. Hong-Jun Yoon, Arvind Ramanathan, Folami Alamudun, and Georgia Tourassi. 2018. Deep radiogenomics for predicting clinical phenotypes in invasive breast cancer. In Proceedings of the 14th International Workshop on Breast Imaging (IWBI’18), Vol. 10718. International Society for Optics and Photonics, 107181H.
108. Guoxian Yu, Guoji Zhang, Huzefa Rangwala, Carlotta Domeniconi, and Zhiwen Yu. 2012. Protein function prediction using weak-label learning. In Proceedings of the ACM Conference on Bioinformatics, Computational Biology and Biomedicine (ACM BCB’12). 202--209.
109. Hwanjo Yu. 2005. SVM selective sampling for ranking with application to data retrieval. In Proceedings of the 11th ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD’05). 354--363.
110. Zheng Jun Zha, Xian Sheng Hua, Tao Mei, Jingdong Wang, Guo Jun Qi, and Zengfu Wang. 2008. Joint multi-label multi-instance learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’08). 1--8.
111. Bang Zhang, Yang Wang, and Fang Chen. 2014. Multilabel image classification via high-order label correlation driven active learning. IEEE Trans. Image Process. 23, 3 (2014), 1430--1441.
112. Bang Zhang, Yang Wang, and Wei Wang. 2012. Batch mode active learning for multi-label image classification with informative label correlation mining. In Proceedings of the IEEE Workshop on the Applications of Computer Vision (WACV’12). 401--407.
113. Jing Zhang, Victor S. Sheng, and Jian Wu. 2019. Crowdsourced label aggregation using bilayer collaborative clustering. IEEE Trans. Neur. Netw. Learn. Syst. 30, 10 (2019), 3172--3185.
114. Xiaoyu Zhang, Jian Cheng, Changsheng Xu, Hanqing Lu, and Songde Ma. 2009. Multi-view multi-label active learning for image classification. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME’09). 258--261.
115. Yi Zhang. 2010. Multi-task active learning with output constraints. In Proceedings of the 24th AAAI Conference on Artificial Intelligence (AAAI’10). 1--6.
116. Shiquan Zhao, Jian Wu, Victor S. Sheng, Chen Ye, Pengpeng Zhao, and Zhiming Cui. 2015. Weak labeled multi-label active learning for image classification. In Proceedings of the ACM International Conference on Multimedia (ACM MM’15). 1127--1130.
117. Yuhui Zheng, Jeon Byeungwoo, Danhua Xu, Q. M. Jonathan Wu, and Zhang Hui. 2015. Image segmentation by generalized hierarchical fuzzy C-means algorithm. J. Intell. Fuzzy Syst. 28, 2 (2015), 961--973.
118. Zhi Hua Zhou. 2010. Semi-supervised learning by disagreement. Knowl. Inf. Syst. 24, 3 (2010), 415--439.
119. Zhi Hua Zhou, Yu Yin Sun, and Yu Feng Li. 2009. Multi-instance learning by treating instances as non-I.I.D. samples. In Proceedings of the Annual International Conference on Machine Learning (ICML’09). 1249--1256.
120. Zhi Hua Zhou and Min Ling Zhang. 2007. Multi-instance multi-label learning with application to scene classification. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS’07). 1609--1616.
121. Zhi Hua Zhou, Min Ling Zhang, Sheng Jun Huang, and Yu Feng Li. 2012. Multi-instance multi-label learning. Artif. Intell. 176, 1 (2012), 2291--2320.
122. Xiaojin Zhu. 2005. Semi-Supervised Learning Literature Survey. Computer Science Technical Report 1530, University of Wisconsin-Madison.

Index Terms

• Theory of computation

  • Theory and algorithms for application domains

    • Machine learning theory

      • Active learning