Putting active learning into multimedia applications: dynamic definition and refinement of concept classifiers

ABSTRACT

The authors developed an extensible system for video exploitation that puts the user in control to better accommodate novel situations and source material. Visually dense displays of thumbnail imagery in storyboard views are used for shot-based video exploration and retrieval. The user can identify a need for a class of audiovisual detection, adeptly and fluently supply training material for that class, and iteratively evaluate and improve the resulting automatic classification produced via multiple modality active learning and SVM. By iteratively reviewing the output of the classifier and updating the positive and negative training samples with less effort than typical for relevance feedback systems, the user can play an active role in directing the classification process while still needing to truth only a very small percentage of the multimedia data set. Examples are given illustrating the iterative creation of a classifier for a concept of interest to be included in subsequent investigations, and for a concept typically deemed irrelevant to be weeded out in follow-up queries. Filtering and browsing tools making use of existing and iteratively added concepts put the user further in control of the multimedia browsing and retrieval process.

References

1. Ahlberg, C. and Shneiderman, B. Visual Information Seeking: Tight Coupling of Dynamic Query Filters with Starfield Displays. In Proc. CHI '94, ACM Press, 1994, 313--317.
2. Burges, C.J.C. A Tutorial on Support Vector Machines for Pattern Recognition. Data Mining and Knowledge Discovery, 2, 2 (1998), 121--167.
3. Chang, E.Y., Tong, S., and Goh, K.-S. Support Vector Machine Concept-Dependent Active Learning for Image Retrieval. IEEE Transactions on Multimedia (anticipated 2005), http://mmdb2.ece.ucsb.edu/~echang/mm000540.pdf.
4. Chang, S.-F., moderator. Multimedia Access and Retrieval: The State of the Art and Future Directions. In Proc. ACM Multimedia '99 (Orlando FL, Nov. 1999), ACM Press, 443--445.
5. Christel, M. and Conescu, R. Addressing the Challenge of Visual Information Access from Digital Image and Video Libraries. In Proc JCDL '05, ACM Press, 2005, 69--78.
6. Forsyth, D., and Ponce, J. Computer Vision: A Modern Approach. Prentice Hall, Englewood Cliffs, NJ, 2002.
7. Freund, Y., and Schapire, R.E. A Decision-Theoretic Generalization of On-line Learning and an Application to Boosting. Journal of Computer and System Sciences, 55, 1, 1997, 119--139.
8. Gosselin, P.H., and Cord, M. RETIN AL: An active learning strategy for image category retrieval. In Proc. IEEE Conf. Image Processing (Singapore, October 2004), 2219--2222.
9. Hastie, T., Tibshirani, R., and Friedman, J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2001.
10. Hauptmann, A.G., and Christel, M.G. Successful Approaches in the TREC Video Retrieval Evaluations. Proc. ACM Multimedia '04, ACM Press (2004), 668--675.
11. Lee, H. and Smeaton, A.F. Designing the User Interface for the Fischlar Digital Video Library, J. Digital Info. 2(4), http://jodi.ecs.soton.ac.uk/Articles/v02/i04/Lee/, May 2002.
12. McCallum, A., and Nigam, K. Employing EM in pool-based active learning for text classification. In Proc. Int'l Conf. on Machine Learning. Morgan Kaufmann, 1998, 350--358.
13. Naphade, M., and Smith, J.R. Active Learning for Simultaneous Annotation of Multiple Binary Concepts. In Proc. IEEE Intl. Conf. on Multimedia and Expo (ICME) (Taipei, Taiwan, June, 2004), 77--80.
14. Naphade, M.R., and Smith, J.R. On the Detection of Semantic Concepts at TRECVID. Proc. ACM Multimedia '04, ACM Press (2004), 660--667.
15. Nguyen, H.T., and Smeulders, A. Active Learning Using Pre-clustering. In Proc. Int'l Conf. on Machine Learning (Banff, Canada, July 2004). ACM Press, 2004.
16. Rowe, L.A. and Jain, R., ACM SIGMM Retreat Report on Future Directions in Multimedia Research, http://www.sigmm.org/Events/reports/retreat03/sigmm-retreat03-final.pdf, March, 2004.
17. Schneiderman, H., and Kanade, T. Probabilistic Modeling of Local Appearance and Spatial Relationships of Object Recognition. In Conf. Computer Vision and Pattern Recognition (CVPR '98) (Santa Barbara, CA, June, 1998). IEEE Computer Society, 1998, 45--51.
18. Tong, S., and Chang, E. Support Vector Machine Active Learning for Image Retrieval. In Proc. ACM Multimedia 2001 (Ottawa, Canada, October, 2001). ACM Press, 2001, 107--118.
19. Trant, J. Image Retrieval Benchmark Database Service: A Needs Assessment and Preliminary Develoment Plan. Council on Library and Information Resources and the Coalition for Networked Information, Archives & Museum Informatics, http://www.clir.org/pubs/reports/ trant04/tranttext.pdf, January 2004.
20. Wang, L., Chan, K.L., and Zhang, Z. Bootstrapping SVM Active Learning by Incorporating Unlabelled Images for Image Retrieval. In Conf. Computer Vision and Pattern Recognition (CVPR '03) (Madison, WI, June, 1998). IEEE Computer Society, 2003, 629--634.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Video segmentation

      • Computer vision tasks

        • Video summarization

      • Image and video acquisition