skip to main content
research-article

Intelligent Traffic Signal Control Based on Reinforcement Learning with State Reduction for Smart Cities

Published:22 July 2021Publication History
Skip Abstract Section

Abstract

Efficient signal control at isolated intersections is vital for relieving congestion, accidents, and environmental pollution caused by increasing numbers of vehicles. However, most of the existing studies not only ignore the constraint of the limited computing resources available at isolated intersections but also the matching degree between the signal timing and the traffic demand, leading to high complexity and reduced learning efficiency. In this article, we propose a traffic signal control method based on reinforcement learning with state reduction. First, a reinforcement learning model is established based on historical traffic flow data, and we propose a dual-objective reward function that can reduce vehicle delay and improve the matching degree between signal time allocation and traffic demand, allowing the agent to learn the optimal signal timing strategy quickly. Second, the state and action spaces of the model are preliminarily reduced by selecting a proper control phase combination; then, the state space is further reduced by eliminating rare or nonexistent states based on the historical traffic flow. Finally, a simplified Q-table is generated and used to optimize the complexity of the control algorithm. The results of simulation experiments show that our proposed control algorithm effectively improves the capacity of isolated intersections while reducing the time and space costs of the signal control algorithm.

References

  1. Monireh Abdoos, Nasser Mozayani, and Ana L. C. Bazzan. 2011. Traffic light control in non-stationary environments based on multi agent Q-learning. In 14th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 1580–1585.Google ScholarGoogle Scholar
  2. Rahib H. Abiyev, Mohammad Ma’aitah, and Bengi Sonyel. 2017. Fuzzy logic traffic lights control (FLTLC). In 9th International Conference on Education Technology and Computers. 233–238. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Sahar Araghi, Abbas Khosravi, Michael Johnstone, and Doug Creighton. 2013. Q-learning method for controlling traffic signal phase time in a single intersection. In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 1261–1265.Google ScholarGoogle ScholarCross RefCross Ref
  4. Bhushan S. Atote, Mangesh Bedekar, and Suja S. Panicker. 2016. Centralized approach towards intelligent traffic signal control. In 2nd International Conference on Information and Communication Technology for Competitive Strategies (ICTCS’16). Association for Computing Machinery, New York, NY. DOI:DOI:https://doi.org/10.1145/2905055.2905121 Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Saif Islam Bouderba and Najem Moussa. 2019. Reinforcement learning (Q-LEARNING) traffic light controller within intersection traffic system. In 4th International Conference on Big Data and Internet of Things. 1–6. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. D. S. Broomhead and D. Lowe. 1988. Multivariable functional interpolation and adaptive networks, complex systems. Complex Systems2 (1998), 321–355. https://www.bibsonomy.org/bibtex/24ef3a0adaabe7e13dcdeee339068f840/mcdiaz.Google ScholarGoogle Scholar
  7. Wentao Chen, Tehuan Chen, and Guang Lin. 2019. Reinforcement learning for traffic control with adaptive horizon. arXiv preprint arXiv:1903.12348 (2019).Google ScholarGoogle Scholar
  8. Lucas Barcelos De Oliveira and Eduardo Camponogara. 2010. Multi-agent model predictive control of signaling split in urban traffic networks. Transport. Res. Part C: Emerg. Technol. 18, 1 (2010), 120–139.Google ScholarGoogle ScholarCross RefCross Ref
  9. Kai Dou, Bin Guo, and Li Kuang. 2019. A privacy-preserving multimedia recommendation in the context of social network based on weighted noise injection. Multimedia Tools Applic. 78, 19 (2019), 26907–26926.Google ScholarGoogle ScholarCross RefCross Ref
  10. Samah El-Tantawy, Baher Abdulhai, and Hossam Abdelgawad. 2013. Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): Methodology and large-scale application on downtown Toronto. IEEE Trans. Intell. Transport. Syst. 14, 3 (2013), 1140–1150. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. Sébastien Faye, Claude Chaudet, and Isabelle Demeure. 2012. A distributed algorithm for multiple intersections adaptive traffic lights control using a wireless sensor networks. In 1st Workshop on Urban Networking. 13–18. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Honghao Gao, Yucong Duan, Lixu Shao, and Xiaobing Sun. 2019. Transformation-based processing of typed resources for multimedia sources in the IoT environment. Wireless Networks (2019), 1–17.Google ScholarGoogle Scholar
  13. Honghao Gao, Wanqiu Huang, and Xiaoxian Yang. 2019. Applying probabilistic model checking to path planning in an intelligent transportation system using mobility trajectories and their statistical data. Intell. Automat. Soft Comput. 25, 3 (2019), 547–559.Google ScholarGoogle Scholar
  14. H. Gao, C. Liu, Y. Li, and X. Yang. 2021. V2VR: Reliable Hybrid-Network-Oriented V2V Data Transmission and Routing Considering RSUs and Connectivity Probability, IEEE Trans. Intell. Transport. Syst. 22, 6 (2021), 3533–3546. DOI:10.1109/TITS.2020.2983835Google ScholarGoogle ScholarCross RefCross Ref
  15. H. Gao, Y. Xu, Y. Yin, W. Zhang, R. Li, and X. Wang. 2020. Context-aware QoS prediction with neural collaborative filtering for Internet-of-Things services. IEEE Internet Things J. 7, 5 (2020), 4532–4542.Google ScholarGoogle ScholarCross RefCross Ref
  16. Wade Genders and Saiedeh Razavi. 2016. Using a deep reinforcement learning agent for traffic signal control. arXiv preprint arXiv:1611.01142 (2016).Google ScholarGoogle Scholar
  17. P. B. Hunt, D. I. Robertson, R. D. Bretherton, and M. Royle. 1982. The SCOOT on-line traffic signal optimisation technique. Traff. Eng. Contr. 23, 4 (1982).Google ScholarGoogle Scholar
  18. Celine Jacob and Baher Abdulhai. 2006. Automated adaptive traffic corridor control using reinforcement learning: Approach and case studies. Transport. Res. Rec. 1959, 1 (2006), 1–8.Google ScholarGoogle ScholarCross RefCross Ref
  19. Li Kuang, Chunbo Hua, Jiagui Wu, Yuyu Yin, and Honghao Gao. 2020. Traffic volume prediction based on multi-sources GPS trajectory data by temporal convolutional network. Mob. Netw. Applic. 25, 4 (2020), 1405–1417.Google ScholarGoogle ScholarCross RefCross Ref
  20. Li Kuang, Xuejin Yan, Xianhan Tan, Shuqi Li, and Xiaoxian Yang. 2019. Predicting taxi demand based on 3D convolutional neural network and multi-task learning. Rem. Sens. 11, 11 (2019), 1265.Google ScholarGoogle ScholarCross RefCross Ref
  21. Daniyar Kurmankhojayev, Gulnur Tolebi, and Nurlan S. Dairbekov. 2019. Road traffic demand estimation and traffic signal control. In 5th International Conference on Engineering and MIS (ICEMIS’19). Association for Computing Machinery, New York, NY.DPO: DOI:https://doi.org/10.1145/3330431.3330433 Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Zhifang Liao, Dayu He, Zhijie Chen, Xiaoping Fan, Yan Zhang, and Shengzong Liu. 2018. Exploring the characteristics of issue-related behaviors in github using visualization techniques. IEEE Access 6 (2018), 24003–24015.Google ScholarGoogle ScholarCross RefCross Ref
  23. P. R. Lowrie. 1990. Scats, Sydney Co-ordinated Adaptive Traffic System: A Traffic Responsive Method Of Controlling Urban Traffic. Technical report.Google ScholarGoogle Scholar
  24. Pei Luo, Qian Ma, and Hui-xian Huang. 2009. Urban trunk road traffic signal coordinated control based on multi-objective immune algorithm. In International Asia Conference on Informatics in Control, Automation and Robotics. IEEE, 72–76. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. Jinming Ma and Feng Wu. 2020. Feudal multi-agent deep reinforcement learning for traffic signal control. In 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS’20). International Foundation for Autonomous Agents and Multiagent Systems, 816–824. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Sandeep Mehan and Vandana Sharma. 2011. Development of traffic light control system based on fuzzy logic. In International Conference on Advances in Computing and Artificial Intelligence (ACAI’11). Association for Computing Machinery, New York, NY, 162–165.DOI:DOI:https://doi.org/10.1145/2007052.2007085 Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Alok Patel, Jayendran Venkateswaran, and Tom V. Mathew. 2015. Optimal signal control for pre-timed signalized junctions with uncertain traffic: Simulation based optimization approach. In Winter Simulation Conference (WSC’15). IEEE Press, 3168–3169. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. K. J. Prabuchandran, Hemanth Kumar A. N., and Shalabh Bhatnagar. 2014. Multi-agent reinforcement learning for traffic signal control. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC’14). IEEE, 2529–2534.Google ScholarGoogle ScholarCross RefCross Ref
  29. Luo Qin, Hou Yufei, and Wang Zhuoqun. 2018. Signal timing simulation of single intersection based on fuzzy-genetic algorithm. In 10th International Conference on Computer Modeling and Simulation (ICCMS’18). Association for Computing Machinery, New York, NY, 28–32.DOI:DOI:https://doi.org/10.1145/3177457.3177496 Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Nouha Rida and Aberrahim Hasbi. 2018. Traffic lights control using wireless sensors networks. In 3rd International Conference on Smart City Applications (SCA’18). Association for Computing Machinery, New York, NY.DOI:DOI:https://doi.org/10.1145/3286606.3286791 Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. The MIT Press. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Thomas L. Thorpe and Charles W. Anderson. 1996. Traffic Light Control Using Sarsa with Three State Representations. Technical Report. IBM corporation.Google ScholarGoogle Scholar
  33. Kazuhiro Tobita and Takashi Nagatani. 2013. Green-wave control of an unbalanced two-route traffic system with signals. Phys. A: Statist. Mech. Applic. 392, 21 (2013), 5422–5430.Google ScholarGoogle ScholarCross RefCross Ref
  34. Elise Van der Pol and Frans A. Oliehoek. 2016. Coordinated deep reinforcement learners for traffic light control. In Workshop on Learning, Inference and Control of Multi-agent Systems (at NIPS 2016).Google ScholarGoogle Scholar
  35. Hong Wang, Chieh Ross Wang, Meixin Zhu, and Wanshi Hong. 2019. Globalized modeling and signal timing control for large-scale networked intersections. In 2nd ACM/EIGSCC Symposium on Smart Cities and Communities (SCC’19). Association for Computing Machinery, New York, NY.DOI:DOI:https://doi.org/10.1145/3357492.3358635 Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Christopher J. C. H. Watkins and Peter Dayan. 1992. Q-learning. Mach. Learn. 8, 3-4 (1992), 279–292.Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Christopher John Cornish Hellaby Watkins. 1989. Learning from Delayed Rewards. PhD Thesis, University of Cambridge, England.Google ScholarGoogle Scholar
  38. Brian Wolshon and William C. Taylor. 1999. Analysis of intersection delay under real-time adaptive signal control. Transport. Res. Part C: Emerg. Technol. 7, 1 (1999), 53–72.Google ScholarGoogle ScholarCross RefCross Ref
  39. Wang Yaping and Zhang Zheng. 2011. A method of reinforcement learning based automatic traffic signal control. In 3rd International Conference on Measuring Technology and Mechatronics Automation, Vol. 1. IEEE, 119–122. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. Yuyu Yin, Lu Chen, Yueshen Xu, Jian Wan, He Zhang, and Zhida Mai. 2019. QoS prediction for service recommendation with deep feature learning in edge computing environment. Mob. Netw. Applic. 25 (2019), 391–401. https://doi.org/10.1007/s11036-019-01241-7Google ScholarGoogle ScholarCross RefCross Ref
  41. Yuyu Yin, Jing Xia, Yu Li, Wenjian Xu, Lifeng Yu, et al. 2019. Group-wise itinerary planning in temporary mobile social network. IEEE Access 7 (2019), 83682–83693.Google ScholarGoogle ScholarCross RefCross Ref
  42. Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. 2019. Multimodal transformer with multi-view visual representation for image captioning. IEEE Trans. Circ. Syst. Vid. Technol. 30, 12 (2019), 4467–4480.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Jun Yu, Min Tan, Hongyuan Zhang, Dacheng Tao, and Yong Rui. 2019. Hierarchical deep click feature prediction for fine-grained image recognition. IEEE Trans. Pattern Anal. Mach. Intell. (2019).Google ScholarGoogle ScholarDigital LibraryDigital Library
  44. Jun Yu, Chaoyang Zhu, Jian Zhang, Qingming Huang, and Dacheng Tao. 2019. Spatial pyramid-enhanced NetVLAD with weighted triplet loss for place recognition. IEEE Transactions on Neural Networks and Learning Systems (2019).Google ScholarGoogle Scholar
  45. Bowen Zheng, Chung-Wei Lin, Shinichi Shiraishi, and Qi Zhu. 2019. Design and analysis of delay-tolerant intelligent intersection management. ACM Transactions on Cyber-Physical Systems 4, 1 (2019), 1–27. Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. Guanjie Zheng, Yuanhao Xiong, Xinshi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Li. 2019. Learning phase competition for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM’19). Association for Computing Machinery, New York, NY, USA, 1963–1972.DOI:https://doi.org/10.1145/3357384.3357900 Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Nantao Zheng, Kairou Wang, Weihua Zhan, and Lei Deng. 2019. Targeting virus-host protein interactions: Feature extraction and machine learning approaches. Current Drug Metabolism 20, 3 (2019), 177–184.Google ScholarGoogle ScholarCross RefCross Ref
  48. Dunhao Zhong and Azzedine Boukerche. 2019. Traffic signal control using deep reinforcement learning with multiple resources of rewards. In Proceedings of the 16th ACM International Symposium on Performance Evaluation of Wireless Ad Hoc, Sensor, & Ubiquitous Networks. 23–28. Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Hongge Zhu. 2019. A platoon-based rolling optimization algorithm at isolated intersections in a connected and autonomous vehicle environment. In Proceedings of the 2019 5th International Conference on Computing and Data Engineering (ICCDE’19). Association for Computing Machinery, New York, NY, USA, 41–46. https://doi.org/10.1145/3330530.3330545 Google ScholarGoogle ScholarDigital LibraryDigital Library
  50. Yonghua Zhu, Weilin Zhang, Yihai Chen, and Honghao Gao. 2019. A novel approach to workload prediction using attention-based LSTM encoder-decoder network in cloud environment. EURASIP Journal on Wireless Communications and Networking 2019, 1 (2019), 274.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Intelligent Traffic Signal Control Based on Reinforcement Learning with State Reduction for Smart Cities

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Internet Technology
        ACM Transactions on Internet Technology  Volume 21, Issue 4
        November 2021
        520 pages
        ISSN:1533-5399
        EISSN:1557-6051
        DOI:10.1145/3472282
        • Editor:
        • Ling Lu
        Issue’s Table of Contents

        Copyright © 2021 Association for Computing Machinery.

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 22 July 2021
        • Accepted: 1 August 2020
        • Revised: 1 July 2020
        • Received: 1 March 2020
        Published in toit Volume 21, Issue 4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!