skip to main content
research-article
Open Access

Mapping Road Safety Features from Streetview Imagery: A Deep Learning Approach

Published:14 September 2020Publication History
Skip Abstract Section

Abstract

Each year, an average of around 6 million car accidents occur in the United States. Road safety features (e.g., concrete barriers, metal crash barriers, rumble strips) play an important role in preventing or mitigating vehicle crashes. Accurate maps of road safety features is an important component of safety management systems for federal or state transportation agencies, helping traffic engineers identify locations to invest in safety infrastructure. In current practice, mapping road safety features is largely done manually (e.g., observations on the road or visual interpretation of streetview imagery), which is both expensive and time consuming. In this article, we propose a deep learning approach to automatically map road safety features from streetview imagery. Unlike existing convolutional neural networks that classify each image individually, we propose to further add a recurrent neural network (long short-term memory) to capture geographic context of images (spatial autocorrelation effect along linear road network paths). Evaluations on real-world streetview imagery show that our proposed model outperforms several baseline methods.

References

  1. Vahid Balali, Elizabeth Depwe, and Mani Golparvar-Fard. 2015. Multi-class traffic sign detection and classification using Google Street View images. In Proceedings of the Transportation Research Board 94th Annual Meeting.Google ScholarGoogle Scholar
  2. Vahid Balali, Armin Ashouri Rad, and Mani Golparvar-Fard. 2015. Detection, classification, and mapping of US traffic signs using Google Street View images for roadway inventory management. Visualization in Engineering 3, 1 (2015), 15.Google ScholarGoogle ScholarCross RefCross Ref
  3. Michael R. Bambach, R. J. Mitchell, and Raphael H. Grzebieta. 2013. The protective effect of roadside barriers for motorcyclists. Traffic Injury Prevention 14, 7 (2013), 756--765.Google ScholarGoogle Scholar
  4. James E. Bryden and Jan S. Fortuniewicz. 1986. Traffic barrier performance related to vehicle size and type. Transportation Research Record 1065 (1986), 69--78.Google ScholarGoogle Scholar
  5. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, Los Alamitos, CA, 248--255.Google ScholarGoogle ScholarCross RefCross Ref
  6. Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Erez Lieberman Aiden, and Li Fei-Fei. 2017. Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States. Proceedings of the National Academy of Sciences 114, 50 (2017), 13108--13113.Google ScholarGoogle ScholarCross RefCross Ref
  7. Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18, 5--6 (2005), 602--610.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Raphael Grzebieta, Mike Bambach, and Andrew McIntosh. 2013. Motorcyclist impacts into roadside barriers: Is the European crash test standard comprehensive enough? Transportation Research Record 2377, 1 (2013), 84--91.Google ScholarGoogle Scholar
  9. Gayle S. W. Hagler, Wei Tang, Matthew J. Freeman, David K. Heist, Steven G. Perry, and Alan F. Vette. 2011. Model evaluation of roadside barrier impact on near-road air pollution. Atmospheric Environment 45, 15 (2011), 2522--2530.Google ScholarGoogle Scholar
  10. Hussein H. Jama, Raphael H. Grzebieta, Rena Friswell, and Andrew S. McIntosh. 2011. Characteristics of fatal motorcycle crashes into roadside safety barriers in Australia and New Zealand. Accident Analysis 8 Prevention 43, 3 (2011), 652--660.Google ScholarGoogle Scholar
  11. Zhe Jiang. 2018. A survey on spatial prediction methods. IEEE Transactions on Knowledge and Data Engineering 31, 9 (2018), 1645--1664.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Zhe Jiang, Michael Evans, Dev Oliver, and Shashi Shekhar. 2016. Identifying K primary corridors from urban bicycle GPS trajectories on a road network. Information Systems 57 (2016), 142--159.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Zhe Jiang, Yan Li, Shashi Shekhar, Lian Rampi, and Joseph Knight. 2017. Spatial ensemble learning for heterogeneous geographic data with class ambiguity: A summary of results. In Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. ACM, New York, NY, 23.Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Zhe Jiang and Arpan Man Sainju. 2019. Hidden Markov contour tree: A spatial structured model for hydrological applications. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 804--813.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Zhe Jiang, Arpan Man Sainju, Yan Li, Shashi Shekhar, and Joseph Knight. 2019. Spatial ensemble learning for heterogeneous geographic data with class ambiguity. ACM Transactions on Intelligent Systems and Technology 10, 4 (2019), 43.Google ScholarGoogle Scholar
  16. Zhe Jiang and Shashi Shekhar. 2017. Spatial Big Data Science. Springer.Google ScholarGoogle Scholar
  17. Zhe Jiang, Shashi Shekhar, Pradeep Mohan, Joseph Knight, and Jennifer Corcoran. 2012. Learning spatial decision tree for geographical classification: A summary of results. In Proceedings of the 20th International Conference on Advances in Geographic Information Systems. ACM, New York, NY, 390--393.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Zhe Jiang, Shashi Shekhar, Xun Zhou, Joseph Knight, and Jennifer Corcoran. 2013. Focal-test-based spatial decision tree learning: A summary of results. In Proceedings of the 2013 IEEE 13th International Conference on Data Mining. IEEE, Los Alamitos, CA, 320--329.Google ScholarGoogle Scholar
  19. Zhe Jiang, Shashi Shekhar, Xun Zhou, Joseph Knight, and Jennifer Corcoran. 2014. Focal-test-based spatial decision tree learning. IEEE Transactions on Knowledge and Data Engineering 27, 6 (2014), 1547--1559.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Zhe Jiang, Miao Xie, and Arpan Man Sainju. 2019. Geographical hidden Markov tree. IEEE Transactions on Knowledge and Data Engineering. Early Access. July 23, 2019.Google ScholarGoogle Scholar
  21. Hawzheen Karim, Rolf Magnusson, and Mats Wiklund. 2012. Assessment of injury rates associated with road barrier collision. Procedia—Social and Behavioral Sciences 48 (2012), 52--63.Google ScholarGoogle Scholar
  22. Driver Knowledge. 2013. Car Accident Statistics in the U.S. Retrieved July 16, 2020 from https://www.driverknowledge.com/car-accident-statistics/.Google ScholarGoogle Scholar
  23. Xiaojiang Li, Chuanrong Zhang, Weidong Li, Robert Ricard, Qingyan Meng, and Weixing Zhang. 2015. Assessing street-level urban greenery using Google Street View and a modified green view index. Urban Forestry 8 Urban Greening 14, 3 (2015), 675--685.Google ScholarGoogle Scholar
  24. Yang Liu, Chengjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional LSTM model and inner-attention. arXiv:1605.09090.Google ScholarGoogle Scholar
  25. Pradeep Mohan, Shashi Shekhar, James A. Shine, James P. Rogers, Zhe Jiang, and Nicole Wayant. 2011. A neighborhood graph based approach to regional co-location pattern discovery: A summary of results. In Proceedings of the 19th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. ACM, New York, NY, 122--132.Google ScholarGoogle Scholar
  26. Aibek Musaev, Zhe Jiang, Steven Jones, Pezhman Sheinidashtegol, and Mirbek Dzhumaliev. 2018. Detection of damage and failure events of road infrastructure using social media. In Proceedings of the International Conference on Web Services. 134--148.Google ScholarGoogle Scholar
  27. Congressional Budget Office. 2018. Public Spending on Transportation and Water Infrastructure, 1956 to 2017. Retrieved July 16, 2020 from https://www.cbo.gov/system/files/2018-10/54539-Infrastructure.pdf.Google ScholarGoogle Scholar
  28. Benjamin Romano and Zhe Jiang. 2017. Visualizing traffic accident hotspots based on spatial-temporal network kernel density estimation. In Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems. ACM, New York, NY, 98.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Carlos Roque and João Lourenço Cardoso. 2013. Observations on the relationship between European standards for safety barrier impact severity and the degree of injury sustained. IATSS Research 37, 1 (2013), 21--29.Google ScholarGoogle Scholar
  30. Arpan Man Sainju, Danial Aghajarian, Zhe Jiang, and Sushil K. Prasad. 2018. Parallel grid-based colocation mining algorithms on GPUs for big spatial event data. IEEE Transactions on Big Data 6, 1 (2018), 107--118.Google ScholarGoogle Scholar
  31. Arpan Man Sainju and Zhe Jiang. 2017. Grid-based colocation mining algorithms on GPU for big spatial event data: A summary of results. In Proceedings of the International Symposium on Spatial and Temporal Databases. 263--280.Google ScholarGoogle Scholar
  32. Jennifer D. Schmidt, Ronald K. Faller, Dean L. Sicking, John D. Reid, Karla A. Lechtenberg, Robert W. Bielenberg, Scott K. Rosenbaugh, et al. 2013. Development of a New Energy-Absorbing Roadside/Median Barrier System with Restorable Elastomer Cartridges. Technical Report. Nebraska Department of Roads.Google ScholarGoogle Scholar
  33. Nico Schulte, Michelle Snyder, Vlad Isakov, David Heist, and Akula Venkatram. 2014. Effects of solid barriers on dispersion of roadway emissions. Atmospheric Environment 97 (2014), 286--295.Google ScholarGoogle Scholar
  34. Shashi Shekhar, Zhe Jiang, Reem Ali, Emre Eftelioglu, Xun Tang, Venkata Gunturi, and Xun Zhou. 2015. Spatiotemporal data mining: A computational perspective. ISPRS International Journal of Geo-Information 4, 4 (2015), 2306--2338.Google ScholarGoogle ScholarCross RefCross Ref
  35. Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi. 2017. Inception-v4, Inception-ResNet and the impact of residual connections on learning. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  36. Zheming Tong, Richard W. Baldauf, Vlad Isakov, Parikshit Deshmukh, and K. Max Zhang. 2016. Roadside vegetation barrier designs to mitigate near-road air pollution impacts. Science of the Total Environment 541 (2016), 920--927.Google ScholarGoogle ScholarCross RefCross Ref
  37. Victor J. D. Tsai, Jyun-Han Chen, and Hsun-Sheng Huang. 2016. Traffic sign inventory from Google Street View images. International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences 41 (2016), 243--246.Google ScholarGoogle Scholar
  38. Carlos M. Vieira, Henrique A. Almeida, Irene S. Ferreira, Joel O. Vasco, Paulo J. Bártolo, Rui B. Ruben, and Sérgio P. Santos. 2008. Development of an impact absorber for roadside barriers. In Proceedings of the 7th LS-DYNA Forum.Google ScholarGoogle Scholar
  39. Yuankai Wu and Huachun Tan. 2016. Short-term traffic flow forecasting with spatial-temporal correlation in a hybrid deep learning framework. arXiv:1612.01022.Google ScholarGoogle Scholar
  40. Miao Xie, Zhe Jiang, and Arpan Man Sainju. 2018. Geographical hidden Markov tree for flood extent mapping. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 2545--2554.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, Yanwei Yu, and Zhenhui Li. 2018. Modeling spatial-temporal dynamics for traffic prediction. arXiv:1803.01254.Google ScholarGoogle Scholar
  42. Huaxiu Yao, Fei Wu, Jintao Ke, Xianfeng Tang, Yitian Jia, Siyu Lu, Pinghua Gong, Jieping Ye, and Zhenhui Li. 2018. Deep multi-view spatial-temporal network for taxi demand prediction. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence.Google ScholarGoogle Scholar
  43. Zhuoning Yuan, Xun Zhou, and Tianbao Yang. 2018. Hetero-ConvLSTM: A deep learning approach to traffic accident prediction on heterogeneous spatio-temporal data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 984--992.Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Zhuoning Yuan, Xun Zhou, Tianbao Yang, James Tamerius, and Ricardo Mantilla. 2017. Predicting traffic accidents through heterogeneous urban data: A case study. In Proceedings of the 6th International Workshop on Urban Computing (UrbComp’17), Vol. 14.Google ScholarGoogle Scholar
  45. Junbo Zhang, Yu Zheng, Dekang Qi, Ruiyuan Li, Xiuwen Yi, and Tianrui Li. 2018. Predicting citywide crowd flows using deep spatio-temporal residual networks. Artificial Intelligence 259 (2018), 147--166.Google ScholarGoogle ScholarCross RefCross Ref
  46. Shanghang Zhang, Guanhang Wu, Joao P. Costeira, and Jose M. F. Moura. 2017. FCN-rLSTM: Deep spatio-temporal neural networks for vehicle counting in city cameras. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17).Google ScholarGoogle Scholar
  47. Yaotian Zou, Andrew P. Tarko, Erdong Chen, and Mario A. Romero. 2014. Effectiveness of cable barriers, guardrails, and concrete barrier walls in reducing the risk of injury. Accident Analysis 8 Prevention 72 (2014), 55--65.Google ScholarGoogle Scholar
  48. A. Mehrara Molan, M. Rezapour, and K. Ksaibati. 2019. Investigating the relationship between the type of traffic barrier and crash severity in trucks and light vehicles. J. Traffic. Transp. Eng. (in press).Google ScholarGoogle Scholar

Index Terms

  1. Mapping Road Safety Features from Streetview Imagery: A Deep Learning Approach

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM/IMS Transactions on Data Science
          ACM/IMS Transactions on Data Science  Volume 1, Issue 3
          Special Issue on Urban Computing and Smart Cities
          August 2020
          217 pages
          ISSN:2691-1922
          DOI:10.1145/3424342
          Issue’s Table of Contents

          Copyright © 2020 ACM

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 14 September 2020
          • Accepted: 1 September 2019
          • Revised: 1 August 2019
          • Received: 1 June 2019
          Published in tds Volume 1, Issue 3

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Research
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!