Abstract
Reduction in the cost of Network Cameras along with a rise in connectivity enables entities all around the world to deploy vast arrays of camera networks. Network cameras offer real-time visual data that can be used for studying traffic patterns, emergency response, security, and other applications. Although many sources of Network Camera data are available, collecting the data remains difficult due to variations in programming interface and website structures. Previous solutions rely on manually parsing the target website, taking many hours to complete. We create a general and automated solution for aggregating Network Camera data spread across thousands of uniquely structured web pages. We analyze heterogeneous web page structures and identify common characteristics among 73 sample Network Camera websites (each website has multiple web pages). These characteristics are then used to build an automated camera discovery module that crawls and aggregates Network Camera data. Our system successfully extracts 57,364 Network Cameras from 237,257 unique web pages.
- Arizona 511. 2019. Arizona Cameras | Live Arizona Cameras | AZ 511. Retrieved from https://www.az511.gov/cctv?.Google Scholar
- N. Abd Manap, G. Di Caterina, J. Soraghan, V. Sidharth, and H. Yao. 2010. Face detection and stereo matching algorithms for smart surveillance system with IP cameras. In Proceedings of the 2nd European Workshop on Visual Information Processing (EUVIP'10). 77–81. https://doi.org/10.1109/EUVIP.2010.5699107Google Scholar
- Remigiusz Baran, Tomasz Ruść, and Mariusz Rychlik. 2014. A smart camera for traffic surveillance. In Multimedia Communications, Services, and Security, Andrzej Dziech and Andrzej Czyżewski (Eds.). Springer International Publishing, Cham, 1–15.Google Scholar
- Dan Brickley, Matthew Burgess, and Natasha Noy. 2019. Google dataset search: Building a search engine for datasets in an open web ecosystem. In Proceedings of the World Wide Web Conference (WWW'19). ACM, New York, NY, 1365–1375. https://doi.org/10.1145/3308558.3313685 Google Scholar
Digital Library
- 511 Alberta CA. 2019. 511AB. Retrieved from https://511.alberta.ca/.Google Scholar
- 511 South Carolina. 2019. South Carolina 511. Retrieved from https://www.511sc.org/#.Google Scholar
- W. Chen, P. Chen, W. Lee, and C. Huang. 2008. Design and implementation of a real time video surveillance system with wireless sensor networks. In Proceedings of the IEEE Vehicular Technology Conference (VTC'08). 218–222. https://doi.org/10.1109/VETECS.2008.57Google Scholar
- Zheng Chen, Liu Wenyin, Feng Zhang, Mingjing Li, and Hongjiang Zhang. 2001. Web mining for web image retrieval. J. Amer. Soc. Info. Sci. Technol. 52, 10 (2001), 831–839. https://doi.org/10.1002/asi.1132 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/asi.1132. Google Scholar
Digital Library
- Mountain News Corporation. 2019. Norway Ski Resort Webcams | OnTheSnow. Retrieved from https://www.onthesnow.com/norway/webcams.html.Google Scholar
- Ryan Dailey, Ahmed S. Kaseb, Chandler Brown, Sam Jenkins, Sam Yellin, Fengjian Pan, and Yung-Hsiang Lu. 2017. Creating the world's largest real-time camera network. In Imaging and Multimedia Analytics in a Web and Mobile World. Retrieved from https://engineering.purdue.edu/HELPS/Publications/papers/2017EI.pdf.Google Scholar
- Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 248–255.Google Scholar
Cross Ref
- eisenbahnlivecam. 2019. ELC. Retrieved from http://eisenbahnlivecam.de.Google Scholar
- The Oklahoma State Regents for Higher Education. 2019. Oklahoma traffic :: Advanced Travelers' Information System. Retrieved from http://www.oktraffic.org/.Google Scholar
- K. Gauen, R. Dailey, J. Laiman, Y. Zi, N. Asokan, Y. Lu, G. K. Thiruvathukal, M. Shyu, and S. Chen. 2017. Comparison of visual datasets for machine learning. In Proceedings of the IEEE International Conference on Information Reuse and Integration (IRI'17). 346–355. https://doi.org/10.1109/IRI.2017.59Google Scholar
- GemsNet. 2019. Webcams from around the World—WorldCam. Retrieved from http://worldcam.eu/.Google Scholar
- Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. 2016. MS-Celeb-1M: A dataset and benchmark for large-scale face recognition. Retrieved from http://arxiv.org/abs/1607.08221.Google Scholar
- A. Halevy, P. Norvig, and F. Pereira. 2009. The unreasonable effectiveness of data. IEEE Intell. Syst. 24, 2 (Mar. 2009), 8–12. https://doi.org/10.1109/MIS.2009.36 Google Scholar
Digital Library
- Robert G. Hallowell, Michael P. Matthews, and Paul A. Pisano. 2005. Automated extraction of weather variables from camera imagery. In Proceedings of the Mid-Continent Transportation Research Symposium. Citeseer, 1–13.Google Scholar
- INDOT. 2019. Indiana Real Time Traffic. Retrieved from http://pws.trafficwise.org/pws/.Google Scholar
- Nathan Jacobs, Walker Burgin, Nick Fridrich, Austin Abrams, Kylia Miskell, Bobby H. Braswell, Andrew D. Richardson, and Robert Pless. 2009. The global network of outdoor webcams: Properties and applications. In Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (GIS'09). ACM, New York, NY, 111–120. https://doi.org/10.1145/1653771.1653789 Google Scholar
Digital Library
- Ahmed S. Kaseb, Wenyi Chen, Ganesh Gingade, and Yung-Hsiang Lu. 2015. Worldview and route planning using live public cameras. In Proc. SPIE, Vol. 9408. 94080I–94080I–8.Google Scholar
- South African National Roads Agency SOC Limited. 2019. KwaZulu-Natal Traffic | i-TRAFFIC | South Africa Traffic | Commuter Information. Retrieved from https://www.i-traffic.co.za/region/KwaZulu-Natal.Google Scholar
- Y. Lu, G. K. Thiruvathukal, A. S. Kaseb, K. Gauen, D. Rijhwani, R. Dailey, D. Malik, Y. Huang, S. Aghajanzadeh, and M. Guo. 2019. See the world through network cameras. Computer 52, 10 (Oct. 2019), 30–40. https://doi.org/10.1109/MC.2019.2906841Google Scholar
Digital Library
- Yung-Hsiang Lu, Andrea Cavallaro, Catherine Crump, Gerald Friedland, and Keith Winstein. 2017. Privacy protection in online multimedia. In Proceedings of the 25th ACM International Conference on Multimedia (MM'17). ACM, New York, NY, 457–459. https://doi.org/10.1145/3123266.3133335 Google Scholar
Digital Library
- Yue Mao, Zeyu Zhang, Hua Sun, and Yang Chen. 2018. CitySense: A data collection approach for city computing applications. In Proceedings of the 16th ACM Conference on Embedded Networked Sensor Systems (SenSys'18). ACM, New York, NY, 379–380. https://doi.org/10.1145/3274783.3275192 Google Scholar
Digital Library
- Google Maps. 2019. Google Maps Platform | Google Developers. Retrieved from https://developers.google.com/maps/documentation/.Google Scholar
- Leonardo Millan-Garcia, Gabriel Sanchez-Perez, Mariko Nakano, Karina Toscano-Medina, Hector Perez-Meana, and Luis Rojas-Cardenas. 2012. An early fire detection algorithm using IP cameras. Sensors 12, 5 (2012), 5670–5686.Google Scholar
Cross Ref
- Inc. MongoDB. 2019. Open Source Document Database | MongoDB. Retrieved from https://www.mongodb.com/.Google Scholar
- Madhumita Murgia. 2019. Microsoft quietly deletes largest public face recognition data set. Retrieved from https://www.ft.com/content/7d3e0d6a-87a0-11e9-a028-86cea8523dc2.Google Scholar
- Suman Nath, Jie Liu, and Feng Zhao. 2006. Challenges in building a portal for sensors world-wide. In Proceedings of the 1st Workshop on World-Sensor-Web. Citeseer.Google Scholar
- 511 Nebraska. 2019. Nebraska 511 Travel Information. Retrieved from https://hb.511.nebraska.gov/#cameras/search?layers=cameras&timeFrame=TODAY.Google Scholar
- The City of New York. 2019. NYCDOT—Real Time Traffic Information. Retrieved from https://webcams.nyctmc.org/.Google Scholar
- City of Tallahassee. 2019. Traffic | Traffic. Retrieved from http://www.talgov.com/traffic/traffic.aspx.Google Scholar
- California Department of Transportation. 2019. District 3—Marysville/Sacramento. Retrieved from http://dot.ca.gov/d3/cameras.html.Google Scholar
- Georgia Department of Transportation. 2019. Georgia 511. Retrieved from http://www.511ga.org/#h_inc_ctl&cam_ctl&l_inc_ctl&zoom=4&lat=4013049.71109&lon=-9397806.77578.Google Scholar
- Iowa Department of Transportation. 2019. Iowa Department of Transportation Traveler Information—Low Bandwidth Web—Cameras—Route Selection. Retrieved from https://lb.511ia.org//ialb/cameras/routeselect.jsf.Google Scholar
- Kansas Department of Transportation. 2019. Kandrive. Retrieved from http://www.kandrive.org/KanDrive/roads/cameras.Google Scholar
- Missouri Department of Transportation. 2019. MoDOT Traveler Information Map. Retrieved from http://traveler.modot.org/map/.Google Scholar
- Montana Department of Transportation. 2019. SCAN Web 6.0. Retrieved from http://rwis.mdt.mt.gov.Google Scholar
- New Brunswick Department of Transportation. 2019. New Brunswick Highway Cameras. Retrieved from https://www2.gnb.ca/content/gnb/en/departments/dti/highways_roads/content/highway_cameras.html.Google Scholar
- Oregon Department of Transportation. 2019. Road & Weather Conditions Map | TripCheck—Oregon Traveler Information. Retrieved from https://www.tripcheck.com/.Google Scholar
- Seattle Department of Transportation. 2019. SDOT Travelers Home Page. Retrieved from https://web6.seattle.gov/travelers/.Google Scholar
- 511 Pennsylvania. 2019. Traffic Cameras. Retrieved from https://www.511pa.com/CameraListing.aspx.Google Scholar
- Landesbetrieb Mobilität Rheinland-Pfalz. 2019. Kameras in RLP. Retrieved from http://www.verkehr.rlp.de/?lang=10&menu1=50&menu2=&menu3=.Google Scholar
- Scrapinghub. 2019. Scrapy | A Fast and Powerful Scraping and Web Crawling Framework. Retrieved from https://scrapy.org/.Google Scholar
- ScrapingHub. 2019. Splash. Retrieved from https://scrapinghub.com/splash.Google Scholar
- USC Web Services. 2019. Tommy Cam | About USC. Retrieved from https://about.usc.edu/tommy-cam/.Google Scholar
- New York State. 2019. Traffic Cameras—New York State Thruway. Retrieved from https://www.thruway.ny.gov/travelers/map/text/twytextcameras.cgi.Google Scholar
- U.S. Geological Survey. 2019. HVO Webcams. Retrieved from https://hvo.wr.usgs.gov/cams/.Google Scholar
- SwissWebcams. 2019. Swiss Webcams—English. Retrieved from https://en.swisswebcams.ch/.Google Scholar
- UDOT TRAFFIC. 2019. UDOT TRAFFIC. Retrieved from http://www.utahcommuterlink.com/.Google Scholar
- Houston TranStar. 2019. Houston TranStar Cameras. Retrieved from https://traffic.houstontranstar.org/cctv/transtar/.Google Scholar
- Travel-cam.net. 2019. Webcams—Travel-cam.net. Retrieved from http://travel-cam.net/en/web-cameras.Google Scholar
- Boston University. 2019. Web Cams | Alumni Association. Retrieved from http://www.bu.edu/alumni/benefits-resources/web-cams/.Google Scholar
- Syracuse University. 2019. Maxwell School of Syracuse University. Retrieved from http://www.maxwell.syr.edu/deans.aspx?id=36507225279.Google Scholar
- Google Street View. 2019. Discover Street View and contribute your own imagery to Google Maps.Retrieved from https://www.google.com/streetview/.Google Scholar
- Long Vu, Indranil Gupta, Jin Liang, and Klara Nahrstedt. 2007. Measurement and modeling of a large-scale overlay for multimedia streaming. In Proceedings of the 4th International Conference on Heterogeneous Networking for Quality, Reliability, Security and Robustness Workshops (QSHINE'07). ACM, New York, NY, Article 3, 7 pages. https://doi.org/10.1145/1577222.1577227 Google Scholar
Digital Library
- Webcamsmania. 2019. Free Webcams—Live Online Cams from the World—WebcamsMania. Retrieved from http://www.webcamsmania.com/.Google Scholar
- W. H. Widen. 2008. Smart cameras and the right to privacy. Proc. IEEE 96, 10 (Oct. 2008), 1688–1697. https://doi.org/10.1109/JPROC.2008.928764Google Scholar
Cross Ref
- WSDOT. 2019. WSDOT—Statewide Route Description and Camera List. Retrieved from http://www.wsdot.com/Traffic/routelist.aspx#SR4.Google Scholar
- P. Yuan, B. Zhang, and J. Li. 2008. Semantic concept learning through massive internet video mining. In Proceedings of the IEEE International Conference on Data Mining Workshops. 847–853. https://doi.org/10.1109/ICDMW.2008.114 Google Scholar
Digital Library
Index Terms
Automated Discovery of Network Cameras in Heterogeneous Web Pages
Recommendations
Current challenges in web crawling
ICWE'13: Proceedings of the 13th international conference on Web EngineeringWeb crawling, a process of collecting web pages in an automated manner, is the primary and ubiquitous operation used by a large number of web systems and agents starting from a simple program for website backup to a major web search engine. Due to an ...
A novel crawling algorithm for web pages
AIRS'11: Proceedings of the 7th Asia conference on Information Retrieval TechnologyCrawler is a main component of search engines. In search engines, crawler part is responsible for discovering and downloading web pages. No search engine can cover whole of the web, thus it has to focus on the most valuable web pages. Several Crawling ...
Web Scraper Application for Extracting Scientific Journals Data
ICFNDS 2021: The 5th International Conference on Future Networks & Distributed SystemsSearching for certain subjects of articles that are disseminated throughout scientific journals would be a time-consuming task, as it would necessitate scouring many digital libraries or journal websites. This process can be performed efficiently by ...






Comments