skip to main content
research-article

Automated Discovery of Network Cameras in Heterogeneous Web Pages

Published:15 October 2021Publication History
Skip Abstract Section

Abstract

Reduction in the cost of Network Cameras along with a rise in connectivity enables entities all around the world to deploy vast arrays of camera networks. Network cameras offer real-time visual data that can be used for studying traffic patterns, emergency response, security, and other applications. Although many sources of Network Camera data are available, collecting the data remains difficult due to variations in programming interface and website structures. Previous solutions rely on manually parsing the target website, taking many hours to complete. We create a general and automated solution for aggregating Network Camera data spread across thousands of uniquely structured web pages. We analyze heterogeneous web page structures and identify common characteristics among 73 sample Network Camera websites (each website has multiple web pages). These characteristics are then used to build an automated camera discovery module that crawls and aggregates Network Camera data. Our system successfully extracts 57,364 Network Cameras from 237,257 unique web pages.

References

  1. Arizona 511. 2019. Arizona Cameras | Live Arizona Cameras | AZ 511. Retrieved from https://www.az511.gov/cctv?.Google ScholarGoogle Scholar
  2. N. Abd Manap, G. Di Caterina, J. Soraghan, V. Sidharth, and H. Yao. 2010. Face detection and stereo matching algorithms for smart surveillance system with IP cameras. In Proceedings of the 2nd European Workshop on Visual Information Processing (EUVIP'10). 77–81. https://doi.org/10.1109/EUVIP.2010.5699107Google ScholarGoogle Scholar
  3. Remigiusz Baran, Tomasz Ruść, and Mariusz Rychlik. 2014. A smart camera for traffic surveillance. In Multimedia Communications, Services, and Security, Andrzej Dziech and Andrzej Czyżewski (Eds.). Springer International Publishing, Cham, 1–15.Google ScholarGoogle Scholar
  4. Dan Brickley, Matthew Burgess, and Natasha Noy. 2019. Google dataset search: Building a search engine for datasets in an open web ecosystem. In Proceedings of the World Wide Web Conference (WWW'19). ACM, New York, NY, 1365–1375. https://doi.org/10.1145/3308558.3313685 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. 511 Alberta CA. 2019. 511AB. Retrieved from https://511.alberta.ca/.Google ScholarGoogle Scholar
  6. 511 South Carolina. 2019. South Carolina 511. Retrieved from https://www.511sc.org/#.Google ScholarGoogle Scholar
  7. W. Chen, P. Chen, W. Lee, and C. Huang. 2008. Design and implementation of a real time video surveillance system with wireless sensor networks. In Proceedings of the IEEE Vehicular Technology Conference (VTC'08). 218–222. https://doi.org/10.1109/VETECS.2008.57Google ScholarGoogle Scholar
  8. Zheng Chen, Liu Wenyin, Feng Zhang, Mingjing Li, and Hongjiang Zhang. 2001. Web mining for web image retrieval. J. Amer. Soc. Info. Sci. Technol. 52, 10 (2001), 831–839. https://doi.org/10.1002/asi.1132 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.1132. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Mountain News Corporation. 2019. Norway Ski Resort Webcams | OnTheSnow. Retrieved from https://www.onthesnow.com/norway/webcams.html.Google ScholarGoogle Scholar
  10. Ryan Dailey, Ahmed S. Kaseb, Chandler Brown, Sam Jenkins, Sam Yellin, Fengjian Pan, and Yung-Hsiang Lu. 2017. Creating the world's largest real-time camera network. In Imaging and Multimedia Analytics in a Web and Mobile World. Retrieved from https://engineering.purdue.edu/HELPS/Publications/papers/2017EI.pdf.Google ScholarGoogle Scholar
  11. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248–255.Google ScholarGoogle ScholarCross RefCross Ref
  12. eisenbahnlivecam. 2019. ELC. Retrieved from http://eisenbahnlivecam.de.Google ScholarGoogle Scholar
  13. The Oklahoma State Regents for Higher Education. 2019. Oklahoma traffic :: Advanced Travelers' Information System. Retrieved from http://www.oktraffic.org/.Google ScholarGoogle Scholar
  14. K. Gauen, R. Dailey, J. Laiman, Y. Zi, N. Asokan, Y. Lu, G. K. Thiruvathukal, M. Shyu, and S. Chen. 2017. Comparison of visual datasets for machine learning. In Proceedings of the IEEE International Conference on Information Reuse and Integration (IRI'17). 346–355. https://doi.org/10.1109/IRI.2017.59Google ScholarGoogle Scholar
  15. GemsNet. 2019. Webcams from around the World—WorldCam. Retrieved from http://worldcam.eu/.Google ScholarGoogle Scholar
  16. Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. 2016. MS-Celeb-1M: A dataset and benchmark for large-scale face recognition. Retrieved from http://arxiv.org/abs/1607.08221.Google ScholarGoogle Scholar
  17. A. Halevy, P. Norvig, and F. Pereira. 2009. The unreasonable effectiveness of data. IEEE Intell. Syst. 24, 2 (Mar. 2009), 8–12. https://doi.org/10.1109/MIS.2009.36 Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Robert G. Hallowell, Michael P. Matthews, and Paul A. Pisano. 2005. Automated extraction of weather variables from camera imagery. In Proceedings of the Mid-Continent Transportation Research Symposium. Citeseer, 1–13.Google ScholarGoogle Scholar
  19. INDOT. 2019. Indiana Real Time Traffic. Retrieved from http://pws.trafficwise.org/pws/.Google ScholarGoogle Scholar
  20. Nathan Jacobs, Walker Burgin, Nick Fridrich, Austin Abrams, Kylia Miskell, Bobby H. Braswell, Andrew D. Richardson, and Robert Pless. 2009. The global network of outdoor webcams: Properties and applications. In Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (GIS'09). ACM, New York, NY, 111–120. https://doi.org/10.1145/1653771.1653789 Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Ahmed S. Kaseb, Wenyi Chen, Ganesh Gingade, and Yung-Hsiang Lu. 2015. Worldview and route planning using live public cameras. In Proc. SPIE, Vol. 9408. 94080I–94080I–8.Google ScholarGoogle Scholar
  22. South African National Roads Agency SOC Limited. 2019. KwaZulu-Natal Traffic | i-TRAFFIC | South Africa Traffic | Commuter Information. Retrieved from https://www.i-traffic.co.za/region/KwaZulu-Natal.Google ScholarGoogle Scholar
  23. Y. Lu, G. K. Thiruvathukal, A. S. Kaseb, K. Gauen, D. Rijhwani, R. Dailey, D. Malik, Y. Huang, S. Aghajanzadeh, and M. Guo. 2019. See the world through network cameras. Computer 52, 10 (Oct. 2019), 30–40. https://doi.org/10.1109/MC.2019.2906841Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Yung-Hsiang Lu, Andrea Cavallaro, Catherine Crump, Gerald Friedland, and Keith Winstein. 2017. Privacy protection in online multimedia. In Proceedings of the 25th ACM International Conference on Multimedia (MM'17). ACM, New York, NY, 457–459. https://doi.org/10.1145/3123266.3133335 Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Yue Mao, Zeyu Zhang, Hua Sun, and Yang Chen. 2018. CitySense: A data collection approach for city computing applications. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems (SenSys'18). ACM, New York, NY, 379–380. https://doi.org/10.1145/3274783.3275192 Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Google Maps. 2019. Google Maps Platform | Google Developers. Retrieved from https://developers.google.com/maps/documentation/.Google ScholarGoogle Scholar
  27. Leonardo Millan-Garcia, Gabriel Sanchez-Perez, Mariko Nakano, Karina Toscano-Medina, Hector Perez-Meana, and Luis Rojas-Cardenas. 2012. An early fire detection algorithm using IP cameras. Sensors 12, 5 (2012), 5670–5686.Google ScholarGoogle ScholarCross RefCross Ref
  28. Inc. MongoDB. 2019. Open Source Document Database | MongoDB. Retrieved from https://www.mongodb.com/.Google ScholarGoogle Scholar
  29. Madhumita Murgia. 2019. Microsoft quietly deletes largest public face recognition data set. Retrieved from https://www.ft.com/content/7d3e0d6a-87a0-11e9-a028-86cea8523dc2.Google ScholarGoogle Scholar
  30. Suman Nath, Jie Liu, and Feng Zhao. 2006. Challenges in building a portal for sensors world-wide. In Proceedings of the 1st Workshop on World-Sensor-Web. Citeseer.Google ScholarGoogle Scholar
  31. 511 Nebraska. 2019. Nebraska 511 Travel Information. Retrieved from https://hb.511.nebraska.gov/#cameras/search?layers=cameras&timeFrame=TODAY.Google ScholarGoogle Scholar
  32. The City of New York. 2019. NYCDOT—Real Time Traffic Information. Retrieved from https://webcams.nyctmc.org/.Google ScholarGoogle Scholar
  33. City of Tallahassee. 2019. Traffic | Traffic. Retrieved from http://www.talgov.com/traffic/traffic.aspx.Google ScholarGoogle Scholar
  34. California Department of Transportation. 2019. District 3—Marysville/Sacramento. Retrieved from http://dot.ca.gov/d3/cameras.html.Google ScholarGoogle Scholar
  35. Georgia Department of Transportation. 2019. Georgia 511. Retrieved from http://www.511ga.org/#h_inc_ctl&cam_ctl&l_inc_ctl&zoom=4&lat=4013049.71109&lon=-9397806.77578.Google ScholarGoogle Scholar
  36. Iowa Department of Transportation. 2019. Iowa Department of Transportation Traveler Information—Low Bandwidth Web—Cameras—Route Selection. Retrieved from https://lb.511ia.org//ialb/cameras/routeselect.jsf.Google ScholarGoogle Scholar
  37. Kansas Department of Transportation. 2019. Kandrive. Retrieved from http://www.kandrive.org/KanDrive/roads/cameras.Google ScholarGoogle Scholar
  38. Missouri Department of Transportation. 2019. MoDOT Traveler Information Map. Retrieved from http://traveler.modot.org/map/.Google ScholarGoogle Scholar
  39. Montana Department of Transportation. 2019. SCAN Web 6.0. Retrieved from http://rwis.mdt.mt.gov.Google ScholarGoogle Scholar
  40. New Brunswick Department of Transportation. 2019. New Brunswick Highway Cameras. Retrieved from https://www2.gnb.ca/content/gnb/en/departments/dti/highways_roads/content/highway_cameras.html.Google ScholarGoogle Scholar
  41. Oregon Department of Transportation. 2019. Road & Weather Conditions Map | TripCheck—Oregon Traveler Information. Retrieved from https://www.tripcheck.com/.Google ScholarGoogle Scholar
  42. Seattle Department of Transportation. 2019. SDOT Travelers Home Page. Retrieved from https://web6.seattle.gov/travelers/.Google ScholarGoogle Scholar
  43. 511 Pennsylvania. 2019. Traffic Cameras. Retrieved from https://www.511pa.com/CameraListing.aspx.Google ScholarGoogle Scholar
  44. Landesbetrieb Mobilität Rheinland-Pfalz. 2019. Kameras in RLP. Retrieved from http://www.verkehr.rlp.de/?lang=10&menu1=50&menu2=&menu3=.Google ScholarGoogle Scholar
  45. Scrapinghub. 2019. Scrapy | A Fast and Powerful Scraping and Web Crawling Framework. Retrieved from https://scrapy.org/.Google ScholarGoogle Scholar
  46. ScrapingHub. 2019. Splash. Retrieved from https://scrapinghub.com/splash.Google ScholarGoogle Scholar
  47. USC Web Services. 2019. Tommy Cam | About USC. Retrieved from https://about.usc.edu/tommy-cam/.Google ScholarGoogle Scholar
  48. New York State. 2019. Traffic Cameras—New York State Thruway. Retrieved from https://www.thruway.ny.gov/travelers/map/text/twytextcameras.cgi.Google ScholarGoogle Scholar
  49. U.S. Geological Survey. 2019. HVO Webcams. Retrieved from https://hvo.wr.usgs.gov/cams/.Google ScholarGoogle Scholar
  50. SwissWebcams. 2019. Swiss Webcams—English. Retrieved from https://en.swisswebcams.ch/.Google ScholarGoogle Scholar
  51. UDOT TRAFFIC. 2019. UDOT TRAFFIC. Retrieved from http://www.utahcommuterlink.com/.Google ScholarGoogle Scholar
  52. Houston TranStar. 2019. Houston TranStar Cameras. Retrieved from https://traffic.houstontranstar.org/cctv/transtar/.Google ScholarGoogle Scholar
  53. Travel-cam.net. 2019. Webcams—Travel-cam.net. Retrieved from http://travel-cam.net/en/web-cameras.Google ScholarGoogle Scholar
  54. Boston University. 2019. Web Cams | Alumni Association. Retrieved from http://www.bu.edu/alumni/benefits-resources/web-cams/.Google ScholarGoogle Scholar
  55. Syracuse University. 2019. Maxwell School of Syracuse University. Retrieved from http://www.maxwell.syr.edu/deans.aspx?id=36507225279.Google ScholarGoogle Scholar
  56. Google Street View. 2019. Discover Street View and contribute your own imagery to Google Maps.Retrieved from https://www.google.com/streetview/.Google ScholarGoogle Scholar
  57. Long Vu, Indranil Gupta, Jin Liang, and Klara Nahrstedt. 2007. Measurement and modeling of a large-scale overlay for multimedia streaming. In Proceedings of the 4th International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness Workshops (QSHINE'07). ACM, New York, NY, Article 3, 7 pages. https://doi.org/10.1145/1577222.1577227 Google ScholarGoogle ScholarDigital LibraryDigital Library
  58. Webcamsmania. 2019. Free Webcams—Live Online Cams from the World—WebcamsMania. Retrieved from http://www.webcamsmania.com/.Google ScholarGoogle Scholar
  59. W. H. Widen. 2008. Smart cameras and the right to privacy. Proc. IEEE 96, 10 (Oct. 2008), 1688–1697. https://doi.org/10.1109/JPROC.2008.928764Google ScholarGoogle ScholarCross RefCross Ref
  60. WSDOT. 2019. WSDOT—Statewide Route Description and Camera List. Retrieved from http://www.wsdot.com/Traffic/routelist.aspx#SR4.Google ScholarGoogle Scholar
  61. P. Yuan, B. Zhang, and J. Li. 2008. Semantic concept learning through massive internet video mining. In Proceedings of the IEEE International Conference on Data Mining Workshops. 847–853. https://doi.org/10.1109/ICDMW.2008.114 Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Automated Discovery of Network Cameras in Heterogeneous Web Pages

            Recommendations

            Comments

            Login options

            Check if you have access through your login credentials or your institution to get full access on this article.

            Sign in

            Full Access

            PDF Format

            View or Download as a PDF file.

            PDF

            eReader

            View online with eReader.

            eReader

            HTML Format

            View this article in HTML Format .

            View HTML Format
            About Cookies On This Site

            We use cookies to ensure that we give you the best experience on our website.

            Learn more

            Got it!