Information-theoretic semantic multimedia indexing

ABSTRACT

To solve the problem of indexing collections with diverse text documents, image documents, or documents with both text and images, one needs to develop a model that supports heterogeneous types of documents. In this paper, we show how information theory supplies us with the tools necessary to develop a unique model for text, image, and text/image retrieval. In our approach, for each possible query keyword we estimate a maximum entropy model based on exclusively continuous features that were preprocessed. The unique continuous feature-space of text and visual data is constructed by using a minimum description length criterion to find the optimal feature-space representation (optimal from an information theory point of view). We evaluate our approach in three experiments: only text retrieval, only image retrieval, and text combined with image retrieval.

References

1. A. Amir, J. O. Argillander, M. Campbell, A. Haubold, G. Iyengar, S. Ebabdollahi, F. Kang, M. Naphade, A. Natsev, J. R. Smith, J. Tesic, T. Volkmer, "IBM Research TRECVID-2005 video retrieval system," TREC Video Retrieval Evaluation Workshop, Gaithersburg, MD, USA, 2005.
2. J. Argillander, G. Iyengar, H. Nock, "Semantic annotation of multimedia using maximum entropy models," IEEE Int'l Conf. on Acoustics, Speech, and Signal Processing, Philadelphia, PA, 2005.
3. K. Barnard, D. A. Forsyth, "Learning the semantics of words and pictures," Int'l Conf. on Computer Vision, Vancouver, Canada, 2001.
4. A. Barron, T. Cover, "Minimum complexity density estimation," IEEE Trans. on Information Theory, vol. 37, pp. 1034--1054, 1991.
5. A. Berger, S. Pietra, V. Pietra, "A maximum entropy approach to natural language processing," Computational Linguistics, 1996.
6. D. Blei, M. Jordan, "Modeling annotated data," ACM SIGIR, Toronto, Canada, 2003.
7. G. Carneiro, N. Vasconcelos, "Formulating semantic image annotation as a supervised learning problem," IEEE Conf. on Computer Vision and Pattern Recognition, San Diego, CA, USA, 2005.
8. S. F. Chen, R. Rosenfeld, "A Gaussian prior for smoothing maximum entropy models," Carnegie Mellon University, Pittsburg, PA February 1999.
9. T. M. Cover, J. A. Thomas, Elements of information theory: John Wiley & Sons, 1991.
10. P. Duygulu, K. Barnard, N. de Freitas, D. Forsyth, "Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary," European Conf. on Computer Vision, Copenhagen, Denmark, 2002.
11. S. L. Feng, V. Lavrenko, R. Manmatha, "Multiple Bernoulli relevance models for image and video annotation," IEEE Conf. on Computer Vision and Pattern Recognition, Cambridge, UK, 2004.
12. M. Figueiredo, A. K. Jain, "Unsupervised learning of finite mixture models," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 24, pp. 381--396, 2002.
13. G. Forman, "An Extensive Empirical Study of Feature Selection Metrics for Text Classification," Machine Learning Research, pp. 1289--1305, 2003.
14. T. Hastie, R. Tibshirani, J. Friedman, The elements of statistical learning: Data mining, inference and prediction: Springer, 2001.
15. T. Hofmann, "Probabilistic latent semantic indexing," ACM SIGIR, Berkeley, CA, USA, 1999.
16. P. Howarth, S. Rüger, "Evaluation of texture features for content-based image retrieval," Int'l Conf. on Image and Video Retrieval, Dublin, Ireland, 2004.
17. J. Jeon, V. Lavrenko, R. Manmatha, "Automatic image annotation and retrieval using cross-media relevance models," ACM SIGIR, Toronto, Canada, 2003.
18. J. Jeon, R. Manmatha, "Using maximum entropy for automatic image annotation," Int'l Conf on Image and Video Retrieval, Dublin, Ireland, 2004.
19. T. Joachims, "Text Categorization with Support Vector Machines: Learning with Many Relevant Features," European Conf. on Machine Learning, 1998.
20. V. Lavrenko, R. Manmatha, J. Jeon, "A model for learning the semantics of pictures," Neural Information Processing System Conf., Vancouver, Canada, 2003.
21. S. Lazebnik, C. Schmid, J. Ponce, "A maximum entropy framework for part-based texture and object recognition," Int'l Conf. on Computer Vision, Beijing, China, 2005.
22. D. C. Liu, J. Nocedal, "On the limited memory method for large scale optimization," Mathematical Programming B, vol. 45, pp. 503--528, 1989.
23. J. Magalhães, S. Rüger, "Logistic regression of generic codebooks for semantic image retrieval," Int'l Conf. on Image and Video Retrieval, Phoenix, AZ, USA, 2006.
24. J. Magalhães, S. Rüger, "High-Dimensional Visual Vocabularies for Image Retrieval," ACM SIGIR, Amsterdam, Holland, 2007.
25. R. Malouf, "A comparison of algorithms for maximum entropy parameter estimation," Sixth Conf. on Natural Language Learning, Taipei, Taiwan, 2002.
26. A. McCallum, K. Nigam, "A comparison of event models for naive Bayes text classification," AAAI Workshop on Learning for Text Categorization, 1998.
27. M. R. Naphade, T. S. Huang, "A probabilistic framework for semantic video indexing filtering and retrieval," IEEE Trans. on Multimedia, vol. 3, pp. 141--151, 2001.
28. K. Nigam, J. Lafferty, A. McCallum, "Using Maximum Entropy for Text Classification," IJCAI - Workshop on Machine Learning for Information Filtering, Stockholm, Sweden, 1999.
29. M. J. Pickering, D. Heesch, R. O'Callaghan, S. Rüger, D. Bull, "Video retrieval using global features in keyframes," TREC Text Retrieval Conf., Gaithersburg, USA, 2002.
30. S. D. Pietra, V. D. Pietra, "Inducing features of random fields," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 19, pp. 380--393, 1997.
31. M. F. Porter, "An algorithm for suffix stripping," Program, vol. 14, pp. 130--137, 1980.
32. J. Rissanen, "Modeling by shortest data description," Automatica, vol. 14, pp. 465--471, 1978.
33. G. Salton, C. Buckley, "Term weighting approaches in automatic text retrieval," Information Processing and Management, vol. 24, pp. 513--523, 1988.
34. C. G. M. Snoek, J. C. v. Gemert, T. Gevers, B. Huurnink, D. C. Koelma, M. v. Liempt, O. d. Rooij, K. E. A. v. d. Sande, F. J. Seinstra, A. W. M. Smeulders, A. H. C. Thean, C. J. Veenman, M. Worring, "The MediaMill TRECVID 2006 Semantic Video Search Engine," TREC Video Retrieval Evaluation Workshop, Gaithersburg, MD, USA, 2006.
35. C. G. M. Snoek, M. Worring, J.-M. Geusebroek, D. C. Koelma, F. J. Seinstra, A. W. M. Smeulders, "The semantic pathfinder: using an authoring metaphor for generic multimedia indexing," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 28, pp. 1678--1689, 2006.
36. A. Vailaya, M. Figueiredo, A. K. Jain, H. J. Zhang, "Image classification for content-based indexing," IEEE Trans. on Image Processing, vol. 10, pp. 117--130, 2001.
37. T. Westerveld, A. P. de Vries, "Experimental result analysis for a generative probabilistic image retrieval model," ACM SIGIR, Toronto, Canada, 2003.
38. T. Westerveld, A. P. de Vries, T. Ianeva, L. Boldareva, D. Hiemstra, "Combining information sources for video retrieval," TREC Video Retrieval Evaluation Workshop, Gaithersburg, MD, USA, 2003.
39. Y. Yang, "An Evaluation of Statistical Approaches to Text Categorization," Information Retrieval, pp. 69--90, 1999.
40. Y. Yang, C. G. Chute, "An example-based mapping method for text categorization and retrieval," ACM Trans. on Information Systems, vol. 13, pp. 252--277, 1994.
41. Y. Yang, X. Liu, "A re-examination of text categorization methods," SIGIR, 1999.
42. Y. Yang, J. O. Pedersen, "A Comparative Study on Feature Selection in Text Categorization," Int'l Conf. on Machine Learning, Nashville, Tennessee, USA, 1997.
43. A. Yavlinsky, E. Schofield, S. Rüger, "Automated image annotation using global features and robust nonparametric density estimation," Int'l Conf. on Image and Video Retrieval, Singapore, 2005.
44. T. Zhang, F. J. Oles, "Text Categorization Based on Regularized Linear Classification Methods," Information Retrieval, pp. 5--31, 2001.

Index Terms

• Information systems

  • Information retrieval

    • Document representation