Establishing the utility of non-text search for news video retrieval with real world users

ABSTRACT

TRECVID participants have enjoyed consistent success using storyboard interfaces for shot-based retrieval, as measured by TRECVID interactive search mean average precision (MAP). However, much is lost by only looking at MAP, and especially by neglecting to bring in representatives of the target user communities to conduct such tasks. This paper reports on the use of within-subjects experiments to reduce subject variability and emphasize the examination of specific video search interface features for their effectiveness in interactive retrieval and user satisfaction. A series of experiments is surveyed to illustrate the gradual realization of getting non-experts to utilize non-textual query features through interface adjustments. Notably, the paper explores the use of the search system by government intelligence analysts, concluding that a variety of search methods are useful for news video retrieval and lead to improved satisfaction. This community, dominated by text search system expertise but still new to video and image search, performed better with and favored a system with image and concept query capabilities over an exclusive text-search system. The user study also found that sports topics mean nothing for this user community and tens of relevant shots collected into the answer set are considered enough to satisfy the information need. Lessons learned from these user interactions are reported, with recommendations on both interface improvements for video retrieval systems and enhancing the ecological validity of video retrieval interface evaluations.

References

1. Christel, M. G., and Conescu, R. M. Addressing the Challenge of Visual Information Access from Digital Image and Video Libraries. In Proc. JCDL (Denver, CO, June 2005), 69--78.
2. Christel, M. G., and Conescu, R. M. Mining Novice User Activity with TRECVID Interactive Retrieval Tasks. In LNCS 4071: Proc. Image and Video Retrieval (CIVR) (Tempe, AZ, July 2006), Springer, Berlin, 2006, 21--30.
3. Christel, M., and Moraveji, N. Finding the Right Shots: Assessing Usability and Performance of a Digital Video Library Interface. In Proc. ACM Multimedia (New York, NY, Oct. 2004), ACM Press, New York, 2004, 732--739.
4. Girgensohn, A., Adcock, J., Cooper, M., and Wilcox, L. Interactive Search in Large Video Collections. In CHI '05 Extended Abstracts on Human Factors in Computing Systems, ACM Press, New York, NY, 2005, 1395--1398.
5. Hauptmann, A. G., and Christel, M. G. Successful Approaches in the TREC Video Retrieval Evaluations. In Proc. ACM Multimedia (Oct. 2004), 668--675.
6. Hollink, L., Nguyen, G. P., Koelma, D. C., Schreiber, A. T., and Worring, M. Assessing User Behaviour in News Video Retrieval. IEE Proc. Vision, Image, & Signal Processing 152(6), 2005, 911--918.
7. Marlow, C., Naaman, M., Boyd, D., and Davis, M. HT06, Tagging Paper, Taxonomy, Flickr, Academic Article, To Read. In Proc. ACM Hypertext and Hypermedia (Odense, Denmark, Aug. 2006), ACM Press, New York, NY, 31--40.
8. Naphade, M., Smith, J. R., Tesic, J., Chang, S.-F., Hsu, W., Kennedy, L., Hauptmann, A. and Curtis, J. Large-Scale Concept Ontology for Multimedia. IEEE MultiMedia 13(3), 2006, 86--91.
9. National Institute of Standards and Technology, NIST TREC Video Retrieval Evaluation Online Proceedings, 2001--2006, http://www-nlpir.nist.gov/projects/tvpubs/tv.pubs.org.html.
10. Shneiderman, B., Byrd, D., and Croft, W. B. Clarifying Search: A User-Interface Framework for Text Searches. D-Lib Magazine, 3, 1 (January 1997), http://www.dlib.org.
11. Shneiderman, B., and Plaisant, C. Strategies for Evaluating Information Visualization Tools: Multi-dimensional In-depth Long-term Case Studies. In Proc. BELIV'06 Workshop, Advanced Visual Interfaces Conf. (Venice, May 2006), 1--7.
12. Smeulders, A., Worring, M., Santini, S., Gupta, A., and Jain, R. Content Based Image Retrieval at the End of the Early Years. IEEE Trans. Pattern Analysis and Machine Intelligence 22(12), 2000, 1349--1380.
13. Snoek, C., Worring, M., Koelma, D., and Smeulders, A. Learned Lexicon-Driven Interactive Video Retrieval. In LNCS 4071: Proc. Image and Video Retrieval (CIVR) (Tempe, AZ, July 2006), Springer, Berlin, 2006, 11--20.
14. Snoek, C., Worring, M., Koelma, D., and Smeulders, A. A Learned Lexicon-Driven Paradigm for Interactive Video Retrieval. IEEE Trans. Multimedia 9(2), Feb. 2007, 280--292.
15. von Ahn, L., and Dabbish, L. Labeling Images with a Computer Game. In Proc. ACM CHI (Vienna, Austria, April 2004), ACM Press, New York, NY, 2004, 319--326.
16. Yan, R., and Hauptmann, A. G. Efficient Margin-Based Rank Learning Algorithms for Information Retrieval. In LNCS 4071: Proc. Image and Video Retrieval (CIVR) (Tempe, AZ, July 2006), Springer, Berlin, 2006, 113--122.
17. Yang, M., Wildemuth, B., and Marchionini, G. The Relative Effectiveness of Concept-based Versus Content-based Video Retrieval. In Proc. ACM Multimedia (Oct. 2004), 368--371.

Index Terms

• Applied computing

  • Computers in other domains

    • Digital libraries and archives

• Information systems

  • Information retrieval

  • Information systems applications

    • Digital libraries and archives