Abstract
In this work, we investigate how to reassign the fully annotated labels at image level to those contextually derived semantic regions, namely Label-to-Region (L2R), in a collective manner. Given a set of input images with label annotations, the basic idea of our approach to L2R is to first discover the patch correspondence across images, and then propagate the common labels shared in image pairs to these correlated patches. Specially, our approach consists of following aspects. First, each of the input images is encoded as a Bag-of-Hierarchical-Patch (BOP) for capturing the rich cues at variant scales, and the individual patches are expressed by patch-level feature descriptors. Second, we present a sparse representation formulation for discovering how well an image or a semantic region can be robustly reconstructed by all the other image patches from the input image set. The underlying philosophy of our formulation is that an image region can be sparsely reconstructed with the image patches belonging to the other images with common labels, while the robustness in label propagation across images requires that these selected patches come from very few images. This preference of being sparse at both patch and image level is named bi-layer sparsity prior. Meanwhile, we enforce the preference of choosing larger-size patches in reconstruction, referred to as continuity-biased prior in this work, which may further enhance the reliability of L2R assignment. Finally, we harness the reconstruction coefficients to propagate the image labels to the matched patches, and fuse the propagation results over all patches to finalize the L2R task. As a by-product, the proposed continuity-biased bi-layer sparse representation formulation can be naturally applied to perform image annotation on new testing images. Extensive experiments on three public image datasets clearly demonstrate the effectiveness of our proposed framework in both L2R assignment and image annotation.
- Bertsekas, D. 1999. Nonlinear Programming. Athena Scientific.Google Scholar
- Candes, E., Romberg, J., and Tao, T. 2006. Stable signal recovery from incomplete and inaccurate measurements. Comm. Pure Appl. Mat. 59, 8, 1207--1223.Google Scholar
Cross Ref
- Cao, L. and Li, F. 2007. Spatially coherent latent topic model for concurrent object segmentation and classification. In Proceedings of the IEEE International Conference on Computer Vision. 1--8.Google Scholar
- Chen, Y., Zhu, L., Yuille, A., and Zhang, H. 2008. Unsupervised learning of probabilistic object models (poms) for object classification, segmentation and recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--8.Google Scholar
- Chua, T., Tang, J., Hong, R., Li, H., Luo, Z., and Zheng, Y. 2009. Nus-wide: A real-world web image database from national university of singapore. In Proceedings of the ACM International Conference on Image and Video Retrieval. 8--10. Google Scholar
Digital Library
- Comite, F., Gilleron, R., and Tommasi, M. 2003. Learning multi-label altenating decision tree from texts and data. In Proceedings of the Machine Learning and Data Mining in Pattern Recognition, Lecture Notes in Computer Science, vol. 2734. 251--274. Google Scholar
Digital Library
- Datta, R., Joshi, D., Li, J., and Wang, J. Z. 2008. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40, 2, 1--60. Google Scholar
Digital Library
- Elisseef, A. and Weston, J. 2001. A kernel method for multi-labelled classification. In Proceedings of the Advances in Neural Information Processing Systems. 681--687.Google Scholar
- Fan, R., Chen, P., and Lin, C. 2005. Working set selection using the second order information for training svm. J. Mach. Learn. Res. 6, 1889--1918. Google Scholar
Digital Library
- Felzenszwalb, P. and Huttenlocher, D. 2004. Efficient graph-based image segmentation. Int. J. Comput. Vis. 59, 2, 167--181. Google Scholar
Digital Library
- Feng, S., Manmatha, R., and Lavrenko, V. 2004. Multiple bernoulli relevance models for image and video annotation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1002--1009. Google Scholar
Digital Library
- Fergus, R., Lee, F., Perona, P., and Zisserman, A. 2005. Learning object categories from google's image search. In Proceedings of the IEEE International Conference on Computer Vision. Google Scholar
Digital Library
- Forsyth, D. and Fleck, M. 1997. Body plans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 678--683. Google Scholar
Digital Library
- Fu, W. 1998. Penalized regression: The bridge versus the lasso. J. Comput. Graph. Statist. 7, 397--416.Google Scholar
- Galleguillos, C., Rabinovich, A., and Belongie, S. 2008. Object categorization using co-occurrence, location and appearance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--8.Google Scholar
- Gu, C., Lim, J., Arbelaez, P., and Malik, J. 2009. Recognition using regions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1030--1037.Google Scholar
- Haering, N., Myles, Z., and Lobo, N. 1997. Locating dedicuous trees. In Proceedings of the IEEE Workshop on Contentbased Access of Image and Video Libraries. 18--25. Google Scholar
Digital Library
- Jacob, L., Obozinski, G., and Vert, J.-P. 2009. Group lasso with overlap and graph lasso. In Proceedings of the International Conference on Machine Learning. Google Scholar
Digital Library
- Jeon, J., Lavrenko, V., and Manmatha, R. 2003. Automatic image annotation and retrieval using cross-media relevance models. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval. 119--126. Google Scholar
Digital Library
- Jin, R., Chai, J., and Si, L. 2004. Effective automatic image annotation via a coherent language model and active learning. In Proceedings of the ACM International Conference on Multimedia. 892--899. Google Scholar
Digital Library
- Kang, F., Jin, R., and Sukthankar, R. 2006. Correlated label propagation with application to multi-label learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1719--1726. Google Scholar
Digital Library
- Lavrenko, V., Manmatha, R., and Jeon, J. 2004. A model for learning the semantics of pictures. In Proceedings of the Advances in Neural Information Processing Systems. 553--560.Google Scholar
- Leibe, B., Leonardis, A., and Schiele, B. 2004. Combined object categorization and segmentation with an implicit shape model. In Proceedings of the ECCV Workshop on Statistical Learning in Computer Vision. 17--32.Google Scholar
- Li, L., Socher, R., and Li, F. 2009. Towards total scene understanding: Classification, annotation and segmentation in an automatic framework. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2036--2043.Google Scholar
- Liu, C., Yuen, J., and Torralba, A. 2010a. Sift flow: Dense correspondence across scenes and its applications. Pattern Anal. Mach. Intell. 99, 1--14. Google Scholar
Digital Library
- Liu, J., Wang, B., Li, M., Li, Z., Ma, W., Lu, H., and Ma, S. 2007. Dual cross-media relevance model for image annotation. In Proceedings of the ACM International Conference on Multimedia. 605--614. Google Scholar
Digital Library
- Liu, X., Cheng, B., Yan, S., Tang, J., Chua, T., and Jin, H. 2009. Label to region by bi-layer sparsity priors. In Proceedings of the ACM International Conference on Multimedia. 115--124. Google Scholar
Digital Library
- Liu, X., Feng, J., Yan, S., and Jin, H. 2010b. Image segmentation with patch-pair density priors. In Proceedings of the ACM International Conference on Multimedia. Google Scholar
Digital Library
- Lowe, D. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60, 2, 91--110. Google Scholar
Digital Library
- Nesterov, Y. 2007. Gradient methods for minimizing composite objective function. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Olshausen, B. and Field, D. 1997. Sparse coding with an overcomplete basis set: A strategy employed by v1? Vis. Res. 37, 23, 3311--3325.Google Scholar
Cross Ref
- Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., and Belongie, S. 2007. Objects in context. In Proceedings of the IEEE International Conference on Computer Vision. 1--8.Google Scholar
- Russell, B., Freeman, W., Efros, A., and Zisserman, A. 2006. Using multiple segmentations to discover objects and their extent in image collections. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1605--1614. Google Scholar
Digital Library
- Serre, T., Wolf, L., and Poggio, T. 2005. Object recognition with features inspired by visual cortex. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Google Scholar
Digital Library
- Shen, Y. and Fan, J. 2010. Leveraging loosely-tagged images and inter-object correlations for tag recommendation. In Proceedings of the ACM Multimedia'10. 5--14. Google Scholar
Digital Library
- Shotton, J., Winn, J., Rother, C., and Criminisi, A. 2006. Textonboost: Joint appearance, shape and context modeling for mulit-class object recognition and segmentation. In Proceedings of the European Conference on Computer Vision. 1--15. Google Scholar
Digital Library
- Singhal, A., Luo, J., and Zhu, W. 2003. Probabilistic spatial context models for scene content understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 18--20. Google Scholar
Digital Library
- Szummer, M. and Picard, R. 1998. Indoor-outdoor image classification. In Proceedings of the IEEE International Workshop on Content-Based Access to Image and Video Databases. 42--51. Google Scholar
Digital Library
- Tibshirani, R. 1996. Regression shrinkage and selection via the lasso. J. Roy. Statist. Soci. B 58, 267--288.Google Scholar
Cross Ref
- Tseng, P. 2008. On accelerated proximal gradient methods for convex-concave optimization. submitted to SIAM J. Optimiz.Google Scholar
- Winn, J. and Jojic, N. 2005. Locus: Learning object classes with unsupervised segmentation. In Proceedings of the IEEE International Conference on Computer Vision. 756--773. Google Scholar
Digital Library
- Wright, J., Ganesh, A., Rao, S., Peng, Y., and Ma, Y. 2009a. Robust principal component analysis: exact recovery of corrupted low-rank matrices via convex optimization. J. ACM.Google Scholar
- Wright, J., Yang, A., Ganesh, A., Sastry, S., and Ma, Y. 2009b. Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31, 2, 210--227. Google Scholar
Digital Library
- Yan, S. and Wang, H. 2009. Semi-supervised learning by sparse representation. In Proceedings of the SIAM International Conference on Data Mining. 792--801.Google Scholar
- Yang, J., Yu, K., Gong, Y., and Huang, T. 2000. Linear spatial pyramid matching using sparse coding for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Yuan, J., Li, J., and Zhang, B. 2007. Exploiting spatial context constraints for automatic image region annotation. In Proceedings of the ACM International Conference on Multimedia. 595--604. Google Scholar
Digital Library
- Yuan, X. and Yan, S. 2010. Visual classification with multi-task joint sparse representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Zhang, J. 2006. A probabilistic framework for multi-task learning. Tech. rep., CMU-LTI-06-006.Google Scholar
- Zhang, M. and Zhou, Z. 2007. Ml-knn: A lazy learning approach to multi-label learning. Pattern Recogn. 40, 7, 2038--2048. Google Scholar
Digital Library
Index Terms
Label-to-region with continuity-biased bi-layer sparsity priors
Recommendations
Label to region by bi-layer sparsity priors
MM '09: Proceedings of the 17th ACM international conference on MultimediaIn this work, we investigate how to automatically reassign the manually annotated labels at the image-level to those contextually derived semantic regions. First, we propose a bi-layer sparse coding formulation for uncovering how an image or semantic ...
Automatic Image Annotation Based on Sparse Representation and Multiple Label Learning
ICVRV '12: Proceedings of the 2012 International Conference on Virtual Reality and VisualizationAutomatic image annotation has emerged as an important research topic due to its potential application on both image understanding and web image search. Due to the inherent ambiguity of image-label mapping, the annotation task has become a challenge to ...
Multi-Label Transfer Learning With Sparse Representation
Due to the visually polysemous barrier, videos and images may be annotated by multiple tags. Discovering the correlations among different tags can significantly help predicting precise labels for videos and images. Many of recent studies toward multi-...






Comments