skip to main content
research-article

Finding Social Points of Interest from Georeferenced and Oriented Online Photographs

Published:28 January 2016Publication History
Skip Abstract Section

Abstract

Points of interest are an important requirement for location-based services, yet they are editorially curated and maintained, either professionally or through community. Beyond the laborious manual annotation task, further complications arise as points of interest may appear, relocate, or disappear over time, and may be relevant only to specific communities. To assist, complement, or even replace manual annotation, we propose a novel method for the automatic localization of points of interest depicted in photos taken by people across the world. Our technique exploits the geographic coordinates and the compass direction supplied by modern cameras, while accounting for possible measurement errors due to the variability in accuracy of the sensors that produced them. We statistically demonstrate that our method significantly outperforms techniques from the research literature on the task of estimating the geographic coordinates and geographic footprints of points of interest in various cities, even when photos are involved in the estimation process that do not show the point of interest at all.

Skip Supplemental Material Section

Supplemental Material

References

  1. S. Agarwal, N. Snavely, I. Simon, S. M. Seitz, and R. Szeliski. 2009. Building Rome in a day. In Proceedings of the IEEE International Conference on Computer Vision. 72--79.Google ScholarGoogle Scholar
  2. S. Ahern, M. Naaman, R. Nair, and J. Yang. 2007. World explorer: Visualizing aggregate data from unstructured text in geo-referenced collections. In Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries. 1--10. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. T. Cham, A. Ciptadi, W. Tan, M. Pham, and L. Chia. 2010. Estimating camera pose from a single urban ground-view omnidirectional image and a 2D building outline map. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 366--373.Google ScholarGoogle Scholar
  4. F. Chang, C. Chen, and C. Lu. 2004. A linear-time component-labeling algorithm using contour tracing technique. Computer Vision and Image Understanding 93, 2 (2004), 206--220. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. L. Chen and A. Roy. 2009. Event detection from Flickr data through wavelet-based spatial analysis. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM’09), 523--532. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. D. J. Crandall, L. Backstrom, D. Huttenlocher, and J. Kleinberg. 2009. Mapping the world’s photos. In Proceedings of the IW3C2 International Conference on World Wide Web. 761--770. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. D. J. Crandall, A. Owens, N. Snavely, and D. P. Huttenlocher. 2013. SfM with MRFs: Discrete-continuous optimization for large-scale structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 12 (2013), 2841--2853. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. P. de Boer, D. P. Kroese, S. Mannor, and R. Y. Rubinstein. 2005. A tutorial on the cross-entropy method. Annals of Operations Research 134, 1 (2005), 19--67.Google ScholarGoogle Scholar
  9. M. De Choudhury, M. Feldman, S. Amer-Yahia, N. Golbandi, R. Lempel, and C. Yu. 2010. Automatic construction of travel itineraries using social breadcrumbs. In Proceedings of the ACM Conference on Hypertext and Hypermedia. 35--44. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B 39, 1 (1977), 1--38.Google ScholarGoogle ScholarCross RefCross Ref
  11. C. Doersch, S. Singh, A. Gupta, J. Sivic, and A. A. Efros. 2012. What makes Paris look like Paris? ACM Transactions on Graphics 31, 4 (2012), 1--9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. C. Dwork, R. Kumar, M. Naor, and D. Sivakumar. 2001. Rank aggregation methods for the Web. In Proceedings of the IW3C2 International Conference on World Wide Web. 613--622. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. B. Epshtein, E. Ofek, Y. Wexler, and P. Zhang. 2007. Hierarchical photo organization using geo-relevance. In Proceedings of the ACM International Conference on Geographic Information Systems. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. M. Ester, H. Kriegel, J. Sander, and X. Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the AAAI International Conference on Knowledge Discovery and Data Mining. 226--231.Google ScholarGoogle Scholar
  15. Facebook, Ericsson, and Qualcomm. 2013. A Focus on Efficiency. Technical Report.Google ScholarGoogle Scholar
  16. G. Field. 1845. Chromatics; or, the Analogy, Harmony, and Philosophy of Colours. D. Bogue.Google ScholarGoogle Scholar
  17. J. Hao, G. Wang, B. Seo, and R. Zimmermann. 2014. Point of interest detection and visual distance estimation for sensor-rich video. IEEE Transactions on Multimedia 16, 7 (2014), 1929--1941.Google ScholarGoogle ScholarCross RefCross Ref
  18. J. Hays and A. A. Efros. 2008. IM2GPS: Estimating geographic information from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1--8.Google ScholarGoogle Scholar
  19. J. Heinly, J. L. Schönberger, E. Dunn, and J. Frahm. 2015. Reconstructing the world in six days. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3287--3295.Google ScholarGoogle Scholar
  20. L. Hollenstein and R. S. Purves. 2010. Exploring place through user-generated content: Using Flickr tags to describe city cores. Journal of Spatial Information Science 1 (2010), 21--48.Google ScholarGoogle Scholar
  21. M. Hölzl, R. Neumeier, and G. Ostzermayer. 2013. Analysis of compass sensor accuracy on several mobile devices in an industrial environment. In Proceedings of the International Conference on Computer Aided Systems Theory. 381--389.Google ScholarGoogle Scholar
  22. A. Jaffe, M. Naaman, T. Tassa, and M. Davis. 2006. Generating summaries and visualization for large collections of geo-referenced photographs. In Proceecings of the ACM International Workshop on Multimedia Information Retrieval. 89--98. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. B. Girshick, S. Guadarrama, and T. Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia. 675--678. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. R. S. Kaminsky, N. Snavely, S. M. Seitz, and R. Szeliski. 2009. Alignment of 3D point clouds to overhead images. In Proceedings of the IEEE Workshop on Computer Vision and Pattern Recognition. 63--70.Google ScholarGoogle Scholar
  25. C. F. F. Karney. 2013. Algorithms for geodesics. Journal of Geodesy 87, 1 (2013), 43--55.Google ScholarGoogle ScholarCross RefCross Ref
  26. L. S. Kennedy and M. Naaman. 2008. Generating diverse and representative image search results for landmarks. In Proceedings of the IW3C2 International Conference on World Wide Web. 297--306. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. S. Kisilevich, F. Mansman, and D. A. Keim. 2010. P-DBSCAN: A density based clustering algorithm for exploration and analysis of attractive areas using collections of geo-tagged photos. In Proceedings of the International Conference and Exhibition on Computing for Geospatial Research & Application. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. J. Kosecká and W. Zhang. 2002. Video compass. In Proceedings of the European Conference on Computer Vision. 476--490. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. A. Krizhevsky, I. Sutskever, and G. E. Hinton. 2012. ImageNet classification with deep convolutional neural networks. In Proceedings of the Annual Conference on Advances in Neural Information Processing Systems. 1097--1105.Google ScholarGoogle Scholar
  30. Y. A. Lacerda, R. G. F. Feitosa, G. A. R. M. Esmeraldo, C. de Souza Baptista, and L. B. Marinho. 2012. Compass clustering: A new clustering method for detection of points of interest using personal collections of georeferenced and oriented photographs. In Proceedings of the Brazilian Symposium on Multimedia and the Web. 281--288. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Y. Li, D. J. Crandall, and D. P. Huttenlocher. 2009. Landmark classification in large-scale image collections. In Proceedings of the IEEE International Conference on Computer Vision. 1957--1964.Google ScholarGoogle Scholar
  32. J. Luo, D. Joshi, J. Yu, and A. C. Gallagher. 2011. Geotagging in multimedia and computer vision—A survey. Multimedia Tools and Applications 51, 1 (2011), 187--211. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Z. Luo, H. Li, J. Tang, R. Hong, and T. Chua. 2010. Estimating poses of world’s photos with geographic metadata. In Advances in Multimedia Modeling. 695--700. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. L. Mummidi and J. Krumm. 2008. Discovering points of interest from users’ map annotations. GeoJournal. 72, 3--4 (2008), 215--227.Google ScholarGoogle ScholarCross RefCross Ref
  35. M. Naaman, Y. J. Song, A. Paepcke, and H. Garcia-Molina. 2004. Automatic organization for digital photographs with geographic coordinates. In Proceedings of the ACM/IEEE-CS Joint Conference on Digital Libraries. 53--62. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. L. Ojeda and J. Borenstein. 2000. Experimental results with the KVH C-100 fluxgate compass in mobile robots. In Proceedings of the IASTED International Conference on Robotics and Applications.Google ScholarGoogle Scholar
  37. J. Paek, J. Kim, and R. Govindan. 2010. Energy-efficient rate-adaptive GPS-based positioning for smartphones. In Proceedings of the ACM International Conference on Mobile Systems, Applications, and Services. 299--314. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. S. Papadopoulos, C. Zigkolis, Y. Kompatsiaris, and A. Vakali. 2011. Cluster-based landmark and event detection for tagged photo collections. IEEE Multimedia 18, 1 (2011), 52--63. Google ScholarGoogle ScholarDigital LibraryDigital Library
  39. D. Pelleg and A. W. Moore. 2000. X-means: Extending K-means with efficient estimation of the number of clusters. In Proceedings of the International Conference on Machine Learning. 727--734. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. V. Pihur, S. Datta, and S. Datta. 2007. Weighted rank aggregation of cluster validation measures: A Monte Carlo cross-entropy approach. Bioinformatics 23, 13 (2007), 1607--1615. Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. A. Popescu. 2013. CEA LIST’s participation at MediaEval 2013 placing task. In Working Notes of the MediaEval Benchmarking Initiative for Multimedia Evaluation.Google ScholarGoogle Scholar
  42. A. Popescu, G. Grefenstette, and P. Moëllic. 2009. Mining tourist information from user-supplied collections. In ACM International Conference on Knowledge and Information Management. 1713--1716. Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. A. Popescu and A. Shabou. 2013. Towards precise POI localization with social media. In Proceedings of the ACM International Conference on Multimedia. 573--576. Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. T. Quack, B. Leibe, and L. J. Van Gool. 2008. World-scale mining of objects and events from community photo collections. In Proceedings of the ACM Conference on Image and Video Retrieval. 47--56. Google ScholarGoogle ScholarDigital LibraryDigital Library
  45. A. Rae, V. Murdock, A. Popescu, and H. Bouchard. 2012. Mining the web for points of interest. In Proceedings of the ACM Interational Conference on Research and Development in Information Retrieval. 711--720. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. R. Raguram, C. Wu, J. Frahm, and S. Lazebnik. 2011. Modeling and recognition of landmark image collections using iconic scene graphs. International Journal of Computer Vision 95, 3 (2011), 213--239. Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. T. Rattenbury, N. Good, and M. Naaman. 2007. Towards automatic extraction of event and place semantics from Flickr tags. In Proceedings of the ACM International Conference on Research and Development in Information Retrieval. 103--110. Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. T. Rattenbury and M. Naaman. 2009. Methods for extracting place semantics from Flickr tags. ACM Transactions on the Web 3, 1 (2009). Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. R. Y. Rubinstein and D. P. Kroese. 2005. The Cross-Entropy Method: A Unified Approach to Combinatorial Optimization, Monte-Carlo Simulation, and Machine Learning. Springer. Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. S. Rudinac, A. Hanjalic, and M. Larson. 2011. Finding representative and diverse community contributed images to create visual summaries of geographic areas. In Proceedings of the ACM International Conference on Multimedia. 1109--1112. Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. P. Serdyukov, V. Murdock, and R. van Zwol. 2009. Placing Flickr photos on a map. In Proceedings of the ACM International Conference on Research and Development in Information Retrieval. 484--491. Google ScholarGoogle ScholarDigital LibraryDigital Library
  52. J. T. Smith. 1797. Remarks on Rural Scenery, with Twenty Etchings of Cottages, from Nature; and Some Observations and Precepts Relative to the Picturesque. N. Smith and I. T. Smith.Google ScholarGoogle Scholar
  53. N. Snavely, S. M. Seitz, and R. Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. ACM Transactions on Graphics 25, 3 (2006), 835--846. Google ScholarGoogle ScholarDigital LibraryDigital Library
  54. N. Snavely, S. M. Seitz, and R. Szeliski. 2008. Modeling the world from internet photo collections. International Journal of Computer Vision 80, 2 (2008), 189--210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. M. A. Stephens. 1974. EDF statistics for goodness of fit and some comparisons. Journal of the American Statistical Association 69, 347 (1974), 730--737.Google ScholarGoogle ScholarCross RefCross Ref
  56. H. C. Thode. 2002. Testing for Normality. CRC Press.Google ScholarGoogle Scholar
  57. B. Thomee. 2013. Localization of points of interest from georeferenced and oriented photographs. In Proceedings of the ACM Interntional Workshop on Geotagging and Its Applications in Multimedia. 19--24. Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. O. Van Laere, S. Schockaert, and B. Dhoedt. 2010. Towards automated georeferencing of Flickr photos. In Proceedings of the ACM Workshop on Geographic Information Retrieval. Google ScholarGoogle ScholarDigital LibraryDigital Library
  59. C. Wang, K. Wilson, and N. Snavely. 2013a. Accurate georegistration of point clouds using geographic data. In Proceedings of the International Conference on 3D Vision. 33--40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. G. Wang, Y. Yin, B. Seo, R. Zimmermann, and Z. Shen. 2013b. Orientation data correction with georeferenced mobile videos. In Proceedings of the ACM International Conference on Advances in Geographic Information Systems. 390--393. Google ScholarGoogle ScholarDigital LibraryDigital Library
  61. C. Wu. 2013. Towards linear-time incremental structure from motion. In Proceedings of the IEEE International Conference on 3D Vision. 127--134. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. Y. Yang, Z. Gong, and L. Hou. 2011. Identifying points of interest by self-tuning clustering. In Proceedings of the ACM International Conference on Research and Development in Information Retrieval. 883--892. Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. P. A. Zandbergen. 2008. Positional accuracy of spatial data: Non-normal distributions and a critique of the national standard for spatial data accuracy. Transactions in GIS 12, 1 (2008), 103--130.Google ScholarGoogle ScholarCross RefCross Ref
  64. P. A. Zandbergen and S. J. Barbeau. 2011. Positional accuracy of assisted GPS data from high-sensitivity GPS-enabled mobile phone. Journal of Navigation 64, 3 (2011), 381--399.Google ScholarGoogle ScholarCross RefCross Ref
  65. Y. Zheng, Z. Zha, and T. S. Chua. 2011. Research and applications on georeferenced multimedia: A survey. Multimedia Tools and Applications 51, 1 (2011), 77--98. Google ScholarGoogle ScholarDigital LibraryDigital Library
  66. Y. Zheng, M. Zhao, Y. Song, H. Adam, U. Buddemeier, A. Bissacco, F. Brucher, T. S. Chua, and H. Neven. 2009. Tour the world: Building a web-scale landmark recognition engine. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1085--1092.Google ScholarGoogle Scholar

Index Terms

  1. Finding Social Points of Interest from Georeferenced and Oriented Online Photographs

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!