Abstract
Efficient signal control at isolated intersections is vital for relieving congestion, accidents, and environmental pollution caused by increasing numbers of vehicles. However, most of the existing studies not only ignore the constraint of the limited computing resources available at isolated intersections but also the matching degree between the signal timing and the traffic demand, leading to high complexity and reduced learning efficiency. In this article, we propose a traffic signal control method based on reinforcement learning with state reduction. First, a reinforcement learning model is established based on historical traffic flow data, and we propose a dual-objective reward function that can reduce vehicle delay and improve the matching degree between signal time allocation and traffic demand, allowing the agent to learn the optimal signal timing strategy quickly. Second, the state and action spaces of the model are preliminarily reduced by selecting a proper control phase combination; then, the state space is further reduced by eliminating rare or nonexistent states based on the historical traffic flow. Finally, a simplified Q-table is generated and used to optimize the complexity of the control algorithm. The results of simulation experiments show that our proposed control algorithm effectively improves the capacity of isolated intersections while reducing the time and space costs of the signal control algorithm.
- Monireh Abdoos, Nasser Mozayani, and Ana L. C. Bazzan. 2011. Traffic light control in non-stationary environments based on multi agent Q-learning. In 14th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 1580–1585.Google Scholar
- Rahib H. Abiyev, Mohammad Ma’aitah, and Bengi Sonyel. 2017. Fuzzy logic traffic lights control (FLTLC). In 9th International Conference on Education Technology and Computers. 233–238. Google Scholar
Digital Library
- Sahar Araghi, Abbas Khosravi, Michael Johnstone, and Doug Creighton. 2013. Q-learning method for controlling traffic signal phase time in a single intersection. In 16th International IEEE Conference on Intelligent Transportation Systems (ITSC). IEEE, 1261–1265.Google Scholar
Cross Ref
- Bhushan S. Atote, Mangesh Bedekar, and Suja S. Panicker. 2016. Centralized approach towards intelligent traffic signal control. In 2nd International Conference on Information and Communication Technology for Competitive Strategies (ICTCS’16). Association for Computing Machinery, New York, NY. DOI:DOI:https://doi.org/10.1145/2905055.2905121 Google Scholar
Digital Library
- Saif Islam Bouderba and Najem Moussa. 2019. Reinforcement learning (Q-LEARNING) traffic light controller within intersection traffic system. In 4th International Conference on Big Data and Internet of Things. 1–6. Google Scholar
Digital Library
- D. S. Broomhead and D. Lowe. 1988. Multivariable functional interpolation and adaptive networks, complex systems. Complex Systems2 (1998), 321–355. https://www.bibsonomy.org/bibtex/24ef3a0adaabe7e13dcdeee339068f840/mcdiaz.Google Scholar
- Wentao Chen, Tehuan Chen, and Guang Lin. 2019. Reinforcement learning for traffic control with adaptive horizon. arXiv preprint arXiv:1903.12348 (2019).Google Scholar
- Lucas Barcelos De Oliveira and Eduardo Camponogara. 2010. Multi-agent model predictive control of signaling split in urban traffic networks. Transport. Res. Part C: Emerg. Technol. 18, 1 (2010), 120–139.Google Scholar
Cross Ref
- Kai Dou, Bin Guo, and Li Kuang. 2019. A privacy-preserving multimedia recommendation in the context of social network based on weighted noise injection. Multimedia Tools Applic. 78, 19 (2019), 26907–26926.Google Scholar
Cross Ref
- Samah El-Tantawy, Baher Abdulhai, and Hossam Abdelgawad. 2013. Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): Methodology and large-scale application on downtown Toronto. IEEE Trans. Intell. Transport. Syst. 14, 3 (2013), 1140–1150. Google Scholar
Digital Library
- Sébastien Faye, Claude Chaudet, and Isabelle Demeure. 2012. A distributed algorithm for multiple intersections adaptive traffic lights control using a wireless sensor networks. In 1st Workshop on Urban Networking. 13–18. Google Scholar
Digital Library
- Honghao Gao, Yucong Duan, Lixu Shao, and Xiaobing Sun. 2019. Transformation-based processing of typed resources for multimedia sources in the IoT environment. Wireless Networks (2019), 1–17.Google Scholar
- Honghao Gao, Wanqiu Huang, and Xiaoxian Yang. 2019. Applying probabilistic model checking to path planning in an intelligent transportation system using mobility trajectories and their statistical data. Intell. Automat. Soft Comput. 25, 3 (2019), 547–559.Google Scholar
- H. Gao, C. Liu, Y. Li, and X. Yang. 2021. V2VR: Reliable Hybrid-Network-Oriented V2V Data Transmission and Routing Considering RSUs and Connectivity Probability, IEEE Trans. Intell. Transport. Syst. 22, 6 (2021), 3533–3546. DOI:10.1109/TITS.2020.2983835Google Scholar
Cross Ref
- H. Gao, Y. Xu, Y. Yin, W. Zhang, R. Li, and X. Wang. 2020. Context-aware QoS prediction with neural collaborative filtering for Internet-of-Things services. IEEE Internet Things J. 7, 5 (2020), 4532–4542.Google Scholar
Cross Ref
- Wade Genders and Saiedeh Razavi. 2016. Using a deep reinforcement learning agent for traffic signal control. arXiv preprint arXiv:1611.01142 (2016).Google Scholar
- P. B. Hunt, D. I. Robertson, R. D. Bretherton, and M. Royle. 1982. The SCOOT on-line traffic signal optimisation technique. Traff. Eng. Contr. 23, 4 (1982).Google Scholar
- Celine Jacob and Baher Abdulhai. 2006. Automated adaptive traffic corridor control using reinforcement learning: Approach and case studies. Transport. Res. Rec. 1959, 1 (2006), 1–8.Google Scholar
Cross Ref
- Li Kuang, Chunbo Hua, Jiagui Wu, Yuyu Yin, and Honghao Gao. 2020. Traffic volume prediction based on multi-sources GPS trajectory data by temporal convolutional network. Mob. Netw. Applic. 25, 4 (2020), 1405–1417.Google Scholar
Cross Ref
- Li Kuang, Xuejin Yan, Xianhan Tan, Shuqi Li, and Xiaoxian Yang. 2019. Predicting taxi demand based on 3D convolutional neural network and multi-task learning. Rem. Sens. 11, 11 (2019), 1265.Google Scholar
Cross Ref
- Daniyar Kurmankhojayev, Gulnur Tolebi, and Nurlan S. Dairbekov. 2019. Road traffic demand estimation and traffic signal control. In 5th International Conference on Engineering and MIS (ICEMIS’19). Association for Computing Machinery, New York, NY.DPO: DOI:https://doi.org/10.1145/3330431.3330433 Google Scholar
Digital Library
- Zhifang Liao, Dayu He, Zhijie Chen, Xiaoping Fan, Yan Zhang, and Shengzong Liu. 2018. Exploring the characteristics of issue-related behaviors in github using visualization techniques. IEEE Access 6 (2018), 24003–24015.Google Scholar
Cross Ref
- P. R. Lowrie. 1990. Scats, Sydney Co-ordinated Adaptive Traffic System: A Traffic Responsive Method Of Controlling Urban Traffic. Technical report.Google Scholar
- Pei Luo, Qian Ma, and Hui-xian Huang. 2009. Urban trunk road traffic signal coordinated control based on multi-objective immune algorithm. In International Asia Conference on Informatics in Control, Automation and Robotics. IEEE, 72–76. Google Scholar
Digital Library
- Jinming Ma and Feng Wu. 2020. Feudal multi-agent deep reinforcement learning for traffic signal control. In 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS’20). International Foundation for Autonomous Agents and Multiagent Systems, 816–824. Google Scholar
Digital Library
- Sandeep Mehan and Vandana Sharma. 2011. Development of traffic light control system based on fuzzy logic. In International Conference on Advances in Computing and Artificial Intelligence (ACAI’11). Association for Computing Machinery, New York, NY, 162–165.DOI:DOI:https://doi.org/10.1145/2007052.2007085 Google Scholar
Digital Library
- Alok Patel, Jayendran Venkateswaran, and Tom V. Mathew. 2015. Optimal signal control for pre-timed signalized junctions with uncertain traffic: Simulation based optimization approach. In Winter Simulation Conference (WSC’15). IEEE Press, 3168–3169. Google Scholar
Digital Library
- K. J. Prabuchandran, Hemanth Kumar A. N., and Shalabh Bhatnagar. 2014. Multi-agent reinforcement learning for traffic signal control. In 17th International IEEE Conference on Intelligent Transportation Systems (ITSC’14). IEEE, 2529–2534.Google Scholar
Cross Ref
- Luo Qin, Hou Yufei, and Wang Zhuoqun. 2018. Signal timing simulation of single intersection based on fuzzy-genetic algorithm. In 10th International Conference on Computer Modeling and Simulation (ICCMS’18). Association for Computing Machinery, New York, NY, 28–32.DOI:DOI:https://doi.org/10.1145/3177457.3177496 Google Scholar
Digital Library
- Nouha Rida and Aberrahim Hasbi. 2018. Traffic lights control using wireless sensors networks. In 3rd International Conference on Smart City Applications (SCA’18). Association for Computing Machinery, New York, NY.DOI:DOI:https://doi.org/10.1145/3286606.3286791 Google Scholar
Digital Library
- Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. The MIT Press. Google Scholar
Digital Library
- Thomas L. Thorpe and Charles W. Anderson. 1996. Traffic Light Control Using Sarsa with Three State Representations. Technical Report. IBM corporation.Google Scholar
- Kazuhiro Tobita and Takashi Nagatani. 2013. Green-wave control of an unbalanced two-route traffic system with signals. Phys. A: Statist. Mech. Applic. 392, 21 (2013), 5422–5430.Google Scholar
Cross Ref
- Elise Van der Pol and Frans A. Oliehoek. 2016. Coordinated deep reinforcement learners for traffic light control. In Workshop on Learning, Inference and Control of Multi-agent Systems (at NIPS 2016).Google Scholar
- Hong Wang, Chieh Ross Wang, Meixin Zhu, and Wanshi Hong. 2019. Globalized modeling and signal timing control for large-scale networked intersections. In 2nd ACM/EIGSCC Symposium on Smart Cities and Communities (SCC’19). Association for Computing Machinery, New York, NY.DOI:DOI:https://doi.org/10.1145/3357492.3358635 Google Scholar
Digital Library
- Christopher J. C. H. Watkins and Peter Dayan. 1992. Q-learning. Mach. Learn. 8, 3-4 (1992), 279–292.Google Scholar
Digital Library
- Christopher John Cornish Hellaby Watkins. 1989. Learning from Delayed Rewards. PhD Thesis, University of Cambridge, England.Google Scholar
- Brian Wolshon and William C. Taylor. 1999. Analysis of intersection delay under real-time adaptive signal control. Transport. Res. Part C: Emerg. Technol. 7, 1 (1999), 53–72.Google Scholar
Cross Ref
- Wang Yaping and Zhang Zheng. 2011. A method of reinforcement learning based automatic traffic signal control. In 3rd International Conference on Measuring Technology and Mechatronics Automation, Vol. 1. IEEE, 119–122. Google Scholar
Digital Library
- Yuyu Yin, Lu Chen, Yueshen Xu, Jian Wan, He Zhang, and Zhida Mai. 2019. QoS prediction for service recommendation with deep feature learning in edge computing environment. Mob. Netw. Applic. 25 (2019), 391–401. https://doi.org/10.1007/s11036-019-01241-7Google Scholar
Cross Ref
- Yuyu Yin, Jing Xia, Yu Li, Wenjian Xu, Lifeng Yu, et al. 2019. Group-wise itinerary planning in temporary mobile social network. IEEE Access 7 (2019), 83682–83693.Google Scholar
Cross Ref
- Jun Yu, Jing Li, Zhou Yu, and Qingming Huang. 2019. Multimodal transformer with multi-view visual representation for image captioning. IEEE Trans. Circ. Syst. Vid. Technol. 30, 12 (2019), 4467–4480.Google Scholar
Digital Library
- Jun Yu, Min Tan, Hongyuan Zhang, Dacheng Tao, and Yong Rui. 2019. Hierarchical deep click feature prediction for fine-grained image recognition. IEEE Trans. Pattern Anal. Mach. Intell. (2019).Google Scholar
Digital Library
- Jun Yu, Chaoyang Zhu, Jian Zhang, Qingming Huang, and Dacheng Tao. 2019. Spatial pyramid-enhanced NetVLAD with weighted triplet loss for place recognition. IEEE Transactions on Neural Networks and Learning Systems (2019).Google Scholar
- Bowen Zheng, Chung-Wei Lin, Shinichi Shiraishi, and Qi Zhu. 2019. Design and analysis of delay-tolerant intelligent intersection management. ACM Transactions on Cyber-Physical Systems 4, 1 (2019), 1–27. Google Scholar
Digital Library
- Guanjie Zheng, Yuanhao Xiong, Xinshi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Li. 2019. Learning phase competition for traffic signal control. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM’19). Association for Computing Machinery, New York, NY, USA, 1963–1972.DOI:https://doi.org/10.1145/3357384.3357900 Google Scholar
Digital Library
- Nantao Zheng, Kairou Wang, Weihua Zhan, and Lei Deng. 2019. Targeting virus-host protein interactions: Feature extraction and machine learning approaches. Current Drug Metabolism 20, 3 (2019), 177–184.Google Scholar
Cross Ref
- Dunhao Zhong and Azzedine Boukerche. 2019. Traffic signal control using deep reinforcement learning with multiple resources of rewards. In Proceedings of the 16th ACM International Symposium on Performance Evaluation of Wireless Ad Hoc, Sensor, & Ubiquitous Networks. 23–28. Google Scholar
Digital Library
- Hongge Zhu. 2019. A platoon-based rolling optimization algorithm at isolated intersections in a connected and autonomous vehicle environment. In Proceedings of the 2019 5th International Conference on Computing and Data Engineering (ICCDE’19). Association for Computing Machinery, New York, NY, USA, 41–46. https://doi.org/10.1145/3330530.3330545 Google Scholar
Digital Library
- Yonghua Zhu, Weilin Zhang, Yihai Chen, and Honghao Gao. 2019. A novel approach to workload prediction using attention-based LSTM encoder-decoder network in cloud environment. EURASIP Journal on Wireless Communications and Networking 2019, 1 (2019), 274.Google Scholar
Digital Library
Index Terms
Intelligent Traffic Signal Control Based on Reinforcement Learning with State Reduction for Smart Cities
Recommendations
Study on Traffic Signal Control Based on Q-Learning
FSKD '09: Proceedings of the 2009 Sixth International Conference on Fuzzy Systems and Knowledge Discovery - Volume 03In order to reduce the delay of vehicles passing through junction, the signal timing of agent controlled intersection was optimized by Q-Learning approach. On the basis of fuzzy rule set, the effect of signal control was improved through optimizing the ...
Study on traffic signal control based on Q-learning
FSKD'09: Proceedings of the 6th international conference on Fuzzy systems and knowledge discovery - Volume 3In order to reduce the delay of vehicles passing through junction, the signal timing of agent controlled intersection was optimized by Q-Learning approach. On the basis of fuzzy rule set, the effect of signal control was improved through optimizing the ...
Intelligent Traffic Signal Control Approach Based on Fuzzy-Genetic Algorithm
FSKD '08: Proceedings of the 2008 Fifth International Conference on Fuzzy Systems and Knowledge Discovery - Volume 03Fuzzy theory and machine learning are applied in this paper. Through fuzzy clustering the number of arriving cars, the schemes of signal control are put into knowledge-database in the form of rule-set under different conditions of cars' arriving. The ...






Comments