skip to main content
research-article

Memory recall based video search: Finding videos you have seen before based on your memory

Published:14 February 2014Publication History
Skip Abstract Section

Abstract

We often remember images and videos that we have seen or recorded before but cannot quite recall the exact venues or details of the contents. We typically have vague memories of the contents, which can often be expressed as a textual description and/or rough visual descriptions of the scenes. Using these vague memories, we then want to search for the corresponding videos of interest. We call this “Memory Recall based Video Search” (MRVS). To tackle this problem, we propose a video search system that permits a user to input his/her vague and incomplete query as a combination of text query, a sequence of visual queries, and/or concept queries. Here, a visual query is often in the form of a visual sketch depicting the outline of scenes within the desired video, while each corresponding concept query depicts a list of visual concepts that appears in that scene. As the query specified by users is generally approximate or incomplete, we need to develop techniques to handle this inexact and incomplete specification by also leveraging on user feedback to refine the specification. We utilize several innovative approaches to enhance the automatic search. First, we employ a visual query suggestion model to automatically suggest potential visual features to users as better queries. Second, we utilize a color similarity matrix to help compensate for inexact color specification in visual queries. Third, we leverage on the ordering of visual queries and/or concept queries to rerank the results by using a greedy algorithm. Moreover, as the query is inexact and there is likely to be only one or few possible answers, we incorporate an interactive feedback loop to permit the users to label related samples which are visually similar or semantically close to the relevant sample. Based on the labeled samples, we then propose optimization algorithms to update visual queries and concept weights to refine the search results. We conduct experiments on two large-scale video datasets: TRECVID 2010 and YouTube. The experimental results demonstrate that our proposed system is effective for MRVS tasks.

References

  1. A. Amir, J. Argillandery, et al. 2005. IBM Research TRECVID-2005 video retrieval system. In Proceedings of the TRECVID Workshop.Google ScholarGoogle Scholar
  2. F. R. Bach, G. R. G. Lanckriet, and M. I. Jordan. 2004. Multiple kernel learning, conic duality, and the SMO algorithm. In Proceedings of the International Conference on Machine Learning. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. P. Browne and A. F. Smeaton. 2005. Video retrieval using dialogue, keyframe similarity and video objects. In Proceedings of the International Conference on Image Process 3, 1208--1211.Google ScholarGoogle Scholar
  4. L. Chaisorn, K. W. Wan, et al. 2010. TRECVID 2010 Known-item Search (KIS) task by I2R. In Proceedings of the TRECVID Workshop.Google ScholarGoogle Scholar
  5. S.-F. Chang, W. H. Hsu, W. Jiang, L. S. Kennedy, D. Xu, A. Yanagawa, and E. Zavesky. 2006. Columbia University Trecvid-2006 video search and high-level feature extraction. In Proceedings of the TRECVID Workshop.Google ScholarGoogle Scholar
  6. X. Y. Chen, J. Yuan, et al. 2010. TRECVID 2010 known-item search by NUS. In Proceedings of the TRECVID Workshop.Google ScholarGoogle Scholar
  7. M. D. Fairchild. 2005. Color Appearance Models 2nd Ed. Addison-Wesley.Google ScholarGoogle Scholar
  8. M. R. Hestenes. 1969. Multiplier and gradient methods. J. Optimization Theory Appl. 303--320.Google ScholarGoogle Scholar
  9. W. H. Hsu, L. S. Kennedy, and S. F. Chang. 2007. Reranking methods for visual search. IEEE Trans. Multimedia 14, 14--22. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. W. M. Hu, D. Xie, Z. Y. Fu, W. R. Zeng, and S. Maybank. 2007. Semantic based surveillance video retrieval. IEEE Trans. Image Process 16, 1168--1181. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. W. M. Hu, N. H. Xie, L. Li, and X. L. Zeng. 2011. A survey on visual content-based video indexing and retrieval. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 41, 797--819. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. L. Kennedy and S.-F. Chang. 2010. Visual ontology construction and concept detection for multimedia indexing and retrieval. In Semantic Computing, 155.Google ScholarGoogle Scholar
  13. L. Kennedy, A. P. Natsev, and S.-F. Chang. 2005. Automatic discovery of query-class-dependent models for multimodal search. In Proceedings of the ACM International Conference on Multimedia. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Y. Liu, T. Mei, and X. S. Hua. 2009a. CrowdReranking: Exploring multiple search engines for visual search reranking. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 500--507. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Y. Liu, Tao Mei, X. Q. Wu, and X.-S. Hua. 2009b. Multigraph-based query-independent learning for video search. IEEE Trans. Circuits Syst. Video Technol. 19, 12, 1841--1850. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Y. F. Ma and H. J. Zhang. 2002. Motion texture: A new motion based video representation. In Proceedings of the International Conference on Pattern Recognition. 548--551.Google ScholarGoogle Scholar
  17. C. D. Manning, P. Raghavan, and H. Schtze. 2009. An Introduction to Information Retrieval. Cambridge University Press, Cambridge, UK. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. S.-Y. Neo, J. Zhao, M.-Y. Kan, and T.-S. Chua. 2006. Video retrieval using high level features: Exploiting query matching and confidence-based weighting. In Proceedings of the ACM International Conference on Image and Video Retrieval. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. L. Q. Nie, M. Wang, Z.-J. Zha, G. D. Li, and T.-S. Chua. 2011. Multimedia answering: Enriching Text QA with media information. In Proceedings of the Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 695--704. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. J. Sivic, M. Everingham, and A. Zisserman. 2005. Person spotting: Video shot retrieval for Dace sets. In Proceedings of the International Conference on Image Video Retrieval. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. C. G. M. Snoek, B. Huurnink, et al. 2007. Adding semantics to detectors for video retrieval. IEEE Trans. Multimedia 9, 975--986. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. C. G. M. Snoek, K. E. A. VandeSande, et al. 2008. The MediaMill TRECVID 2008 semantic video search engine. In Proceedings of the TRECVID Workshop.Google ScholarGoogle Scholar
  23. C. G. M. Snoek and M. Worring. 2009. Concept-based video retrieval. In Foundations and Trends in Information Retrieval 2, 215--322. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. TRECVID2010. 2010. TRECVID2010. http://www-nlpir.nist.gov/projects/tv2010/tv2010.html (2010).Google ScholarGoogle Scholar
  25. D. Wang, X. Li, J. Li, and B. Zhang. 2007. The importance of query concept-mapping for automatic video retrieval. In Proceedings of the International Conference on Multimedia. 285--288. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. R. Yan and A. G. Hauptmann. 2007. A review of text and image retrieval approaches for broadcast news video. Inf. Retrieval 10, 445--484. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. A. Yanagawa, S.-F. Chang, L. Kennedy, and W. H. Hsu. 2007. Columbia University's baseline detectors for 374 LSCOM semantic visual concepts. ADVENT Tech. rep. 222-2006-8.Google ScholarGoogle Scholar
  28. J. Yang and A. G. Hauptmann. 2006. Exploring temporal consistency for video analysis and retrieval. In Proceedings of the International Conference on Multimedia Information Retrieval. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. T. Yao, C.-W. Ngo, and T. Mei. 2013. Circular reranking for visual search. IEEE Trans. Image Process. 22, 4, 1644--1655. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. J. Yuan, Z.-J. Zha, Y.-T. Zheng, M. Wang, X. D. Zhou, and T.-S. Chua. 2011a. Learning concept bundles for video search with complex queries. In Proceedings of the ACM International Conference on Multimedia. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. J. Yuan, Z.-J. Zha, Y.-T. Zheng, M. Wang, X. D. Zhou, and T.-S. Chua. 2011b. Utilizing related samples to enhance interactive concept-based video search. IEEE Trans. Multimedia 13, 1343--1355. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. E. Zavesky and S.-F. Chang. 2008. CuZero: Embracing the Frontier of interactive visual search for informed users. In Proceedings of the International Conference on Multimedia Information Retrieval. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Z. J. Zha, L. J. Yang, T. Mei, M. Wang, and Z. F. Wang. 2009. Visual query suggestion. In Proceedings of the International Conference on Multimedia. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Memory recall based video search: Finding videos you have seen before based on your memory

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in

    Full Access

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader
    About Cookies On This Site

    We use cookies to ensure that we give you the best experience on our website.

    Learn more

    Got it!