A novel fusion method for integrating multiple modalities and knowledge for multimodal location estimation

ABSTRACT

This article describes a novel fusion approach using multiple modalities and knowledge sources that improves the accuracy of multimodal location estimation algorithms. The problem of "multimodal location estimation" or "placing" involves associating geo-locations with consumer-produced nmultimedia data like videos or photos that have not been tagged using GPS. Our algorithm effectively integrates data from the visual and textual modalities with external geographical knowledge bases by building a hierarchical model that combines data-driven and semantic methods to group visual and textual features together within geographical regions. We evaluate our algorithm on the MediaEval 2010 Placing Task dataset and show that our system significantly outperforms other state-of-the-art approaches, successfully locating about 40% of the videos to within a radius of 100 m.

References

1. http://translate.google.com.
2. http://www.geonames.org.
3. http://www.wikipedia.org.
4. http://code.google.com/apis/maps/index.html.
5. J. Baldridge. The OpenNLP Project. http://www.opennlp.com, 2005.
6. J. Choi, G. Friedland, V. Ekambaram, and K. Ramchandran. Multimodal location estimation of consumer media: Dealing with sparse training data. In Multimedia and Expo (ICME), 2012 IEEE International Conference on, pages 43--48, July.
7. J. Choi, A. Janin, and G. Friedland. The 2010 ICSI Video Location Estimation System. In Working Notes of the MediaEval 2010 Workshop, 2010.
8. D. Ferrés and H. Rodríguez. Talp at mediaeval 2010 placing task: Geographical focus detection of flickr mtextual annotations. Proceedings of MediaEval, 2010.
9. G. Friedland, O. Vinyals, and T. Darrell. Multimodal Location Estimation. In Proceedings of ACM Multimedia, pages 1245--1251, 2010.
10. J. Hays and A. A. Efros. IM2GPS: estimating geographic information from a single image. In IEEE Conference on Computer Vision and Pattern mRecognition, 2008. CVPR 2008, pages 1--8. IEEE, June 2008.
11. L. A. U.-L. M. G.-V. José M. Perea-Ortega, Miguel A. García-Cumbreras. In Working Notes Proceedings of the MediaEval 2010 Workshop, Pisa, Italy, 2010.
12. P. Kelm, S. Schmiedeke, and T. Sikora. Multi-modal, multi-resource methods for placing flickr videos on the map. In ACM International Conference on Multimedia Retrieval (ICMR), page 8, Apr. 2011.
13. M. Larson, M. Soleymani, P. Serdyukov, S. Rudinac, C. Wartena, V. Murdock, G. Friedland, R. Ordelman, and G. J. F. Jones. Automatic tagging and geotagging in video collections and communities. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval, ICMR '11, pages 51:1--51:8, New York, NY, USA, 2011. ACM.
14. T. Rattenbury and M. Naaman. Methods for extracting place semantics from flickr tags. ACM Trans. Web, 3(1):1:1--1:30, Jan. 2009.
15. O. Van Laere, S. Schockaert, and B. Dhoedt. Ghent University at the 2010 Placing Task. In Proceedings of MediaEval, October 2010.

Index Terms

• Information systems

  • Information retrieval

    • Document representation

    • Retrieval models and ranking

    • Retrieval tasks and goals

      • Clustering and classification

  • Information systems applications

    • Data mining

      • Clustering

• Theory of computation

  • Semantics and reasoning

    • Program reasoning

      • Abstraction