research-article

Open access

Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting

Authors:

Yongjun XuAuthors Info & Claims

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Pages 1567 - 1577

https://doi.org/10.1145/3534678.3539396

Published: 14 August 2022 Publication History

Abstract

Multivariate Time Series (MTS) forecasting plays a vital role in a wide range of applications. Recently, Spatial-Temporal Graph Neural Networks (STGNNs) have become increasingly popular MTS forecasting methods. STGNNs jointly model the spatial and temporal patterns of MTS through graph neural networks and sequential models, significantly improving the prediction accuracy. But limited by model complexity, most STGNNs only consider short-term historical MTS data, such as data over the past one hour. However, the patterns of time series and the dependencies between them (i.e., the temporal and spatial patterns) need to be analyzed based on long-term historical MTS data. To address this issue, we propose a novel framework, in which STGNN is Enhanced by a scalable time series Pre-training model (STEP). Specifically, we design a pre-training model to efficiently learn temporal patterns from very long-term history time series (e.g., the past two weeks) and generate segment-level representations. These representations provide contextual information for short-term time series input to STGNNs and facilitate modeling dependencies between time series. Experiments on three public real-world datasets demonstrate that our framework is capable of significantly enhancing downstream STGNNs, and our pre-training model aptly captures temporal patterns.

Supplemental Material

MP4 File

Presentation video - short version

Download
19.85 MB

References

[1]

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. 2019. MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixing. In ICML.

[2]

Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. 2020. Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting. In NeurIPS.

[3]

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-Shot Learners. In NeurIPS.

[4]

Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Congrui Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, and Qi Zhang. 2020. Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting. In NeurIPS.

[5]

Chao Chen, Karl Petty, Alexander Skabardonis, Pravin Varaiya, and Zhanfeng Jia. 2001. Freeway performance measurement system: mining loop detector data. Transportation Research Record (2001).

[6]

Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. In SSST@EMNLP.

[7]

Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In NeurIPS.

[8]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL.

[9]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR.

[10]

Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. 2019. Learning Discrete Structures for Graph Neural Networks. In ICML.

[11]

Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019. Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting. In AAAI.

[12]

Shengnan Guo, Youfang Lin, Huaiyu Wan, Xiucheng Li, and Gao Cong. 2021. Learning dynamics and heterogeneity of spatial-temporal graph data for traffic forecasting. TKDE (2021).

[13]

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 (2021).

[14]

Hosagrahar V Jagadish, Johannes Gehrke, Alexandros Labrinidis, Yannis Papakonstantinou, Jignesh M Patel, Raghu Ramakrishnan, and Cyrus Shahabi. 2014. Big data and its technical challenges. Commun. ACM (2014).

[15]

Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical Reparameterization with Gumbel-Softmax. In ICLR.

[16]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In ICLR.

[17]

Thomas Kipf, Ethan Fetaya, Kuan-ChiehWang, MaxWelling, and Richard Zemel. 2018. Neural relational inference for interacting systems. In ICML.

[18]

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR.

[19]

Fuxian Li, Jie Feng, Huan Yan, Guangyin Jin, Depeng Jin, and Yong Li. 2021. Dynamic Graph Convolutional Recurrent Network for Traffic Prediction: Benchmark and Solution. CoRR (2021). arXiv:2104.14917

[20]

Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018. Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. In ICLR.

[21]

Haozhe Lin, Yushun Fan, Jia Zhang, and Bing Bai. 2021. REST: Reciprocal Framework for Spatiotemporal-coupled Predictions. In TheWebConference.

[22]

Ilya Loshchilov and Frank Hutter. 2018. DecoupledWeight Decay Regularization. In ICLR.

[23]

Zheng Lu, Chen Zhou, Jing Wu, Hao Jiang, and Songyue Cui. 2016. Integrating Granger Causality and Vector Auto-Regression for Traffic Prediction of Large- Scale WLANs. KSII Trans. Internet Inf. Syst. (2016).

[24]

Helmut Lütkepohl. 2005. New introduction to multiple time series analysis. Springer Science & Business Media.

Digital Library

[25]

Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In ICLR.

[26]

Zheyi Pan, Yuxuan Liang, Weifeng Wang, Yong Yu, Yu Zheng, and Junbo Zhang. 2019. Urban traffic prediction from spatio-temporal data using deep meta learning. In SIGKDD. 1720--1730.

[27]

Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL.

[28]

Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences (2020).

[29]

Chao Shang, Jie Chen, and Jinbo Bi. 2021. Discrete Graph Structure Learning for Forecasting Multiple Time Series. In ICLR.

[30]

Alexander J. Smola and Bernhard Schölkopf. 2004. A tutorial on support vector regression. Stat. Comput. (2004).

[31]

Chao Song, Youfang Lin, Shengnan Guo, and HuaiyuWan. 2020. Spatial-Temporal Synchronous Graph Convolutional Networks: A New Framework for Spatial- Temporal Network Data Forecasting. In AAAI.

[32]

Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In NeurIPS.

Digital Library

[33]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS.

[34]

Xiaoyang Wang, Yao Ma, Yiqi Wang, Wei Jin, Xin Wang, Jiliang Tang, Caiyan Jia, and Jian Yu. 2020. Traffic Flow Prediction via Spatial Temporal Graph Neural Network. In WWW.

[35]

ZonghanWu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. In SIGKDD.

[36]

Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019. Graph WaveNet for Deep Spatial-Temporal Graph Modeling. In IJCAI.

[37]

Yongjun Xu, Xin Liu, Xin Cao, Changping Huang, Enke Liu, Sen Qian, Xingchen Liu, Yanjun Wu, Fengliang Dong, Cheng-Wei Qiu, et al. 2021. Artificial intelligence: A powerful paradigm for scientific research. The Innovation 2, 4 (2021).

[38]

Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018. Spatio-Temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. In IJCAI.

[39]

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A Transformer-based Framework for Multivariate Time Series Representation Learning. In SIGKDD.

[40]

Ling Zhao, Yujiao Song, Chao Zhang, Yu Liu, Pu Wang, Tao Lin, Min Deng, and Haifeng Li. 2020. T-GCN: A Temporal Graph Convolutional Network for Traffic Prediction. IEEE TITS (2020).

[41]

Chuanpan Zheng, Xiaoliang Fan, Cheng Wang, and Jianzhong Qi. 2020. GMAN: A Graph Multi-Attention Network for Traffic Prediction. In AAAI.

Cited By

Zhang HDing KXie JXiao WXie Y(2025)Flow prediction via adaptive dynamic graph with spatio-temporal correlationsExpert Systems with Applications10.1016/j.eswa.2024.125474261(125474)Online publication date: Feb-2025
https://doi.org/10.1016/j.eswa.2024.125474
Du YHuang JChen JChen KWang JHe Q(2024)Enhanced Transformer Framework for Multivariate Mesoscale Eddy Trajectory PredictionJournal of Marine Science and Engineering10.3390/jmse1210175912:10(1759)Online publication date: 4-Oct-2024
https://doi.org/10.3390/jmse12101759
Xia YLu J(2024)Short-Term Flood Prediction Model Based on Pre-Training EnhancementElectronics10.3390/electronics1311220313:11(2203)Online publication date: 5-Jun-2024
https://doi.org/10.3390/electronics13112203
Show More Cited By

Index Terms

Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting
1. Information systems
  1. Information systems applications
    1. Data mining

Recommendations

Spatial-Temporal Identity: A Simple yet Effective Baseline for Multivariate Time Series Forecasting
CIKM '22: Proceedings of the 31st ACM International Conference on Information & Knowledge Management

Multivariate Time Series (MTS) forecasting plays a vital role in a wide range of applications. Recently, Spatial-Temporal Graph Neural Networks (STGNNs) have become increasingly popular MTS forecasting methods due to their state-of-the-art performance. ...
Neural network applications in time series forecasting
Spatial-temporal graph neural network for traffic forecasting: An overview and open research issues
Abstract
Traffic forecasting plays an important role of modern Intelligent Transportation Systems (ITS). With the recent rapid advancement in deep learning, graph neural networks (GNNs) have become an emerging research issue for improving the traffic ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 2022

5033 pages

ISBN:9781450393850

DOI:10.1145/3534678

General Chairs:
Aidong Zhang
University of Virginia
,
Huzefa Rangwala
Amazon/George Mason University

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2022

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

China Postdoctoral Science Foundation
National Science Foundation of China

Conference

KDD '22

Sponsor:

KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 14 - 18, 2022

Washington DC, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

89
Total Citations
View Citations
5,821
Total Downloads

Downloads (Last 12 months)2,847
Downloads (Last 6 weeks)265

Reflects downloads up to 06 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang HDing KXie JXiao WXie Y(2025)Flow prediction via adaptive dynamic graph with spatio-temporal correlationsExpert Systems with Applications10.1016/j.eswa.2024.125474261(125474)Online publication date: Feb-2025
https://doi.org/10.1016/j.eswa.2024.125474
Du YHuang JChen JChen KWang JHe Q(2024)Enhanced Transformer Framework for Multivariate Mesoscale Eddy Trajectory PredictionJournal of Marine Science and Engineering10.3390/jmse1210175912:10(1759)Online publication date: 4-Oct-2024
https://doi.org/10.3390/jmse12101759
Xia YLu J(2024)Short-Term Flood Prediction Model Based on Pre-Training EnhancementElectronics10.3390/electronics1311220313:11(2203)Online publication date: 5-Jun-2024
https://doi.org/10.3390/electronics13112203
Chen PHe WMa WHuang XWang C(2024)IoTDQ: An Industrial IoT Data Analysis Library for Apache IoTDBBig Data Mining and Analytics10.26599/BDMA.2023.90200107:1(29-41)Online publication date: Mar-2024
https://doi.org/10.26599/BDMA.2023.9020010
YE CMA Q(2024)Semantic Relationship-Based Unsupervised Representation Learning of Multivariate Time SeriesIEICE Transactions on Information and Systems10.1587/transinf.2023EDP7046E107.D:2(191-200)Online publication date: 1-Feb-2024
https://doi.org/10.1587/transinf.2023EDP7046
Han JZhang WLiu HTao TTan NXiong H(2024)BigST: Linear Complexity Spatio-Temporal Graph Neural Network for Traffic Forecasting on Large-Scale Road NetworksProceedings of the VLDB Endowment10.14778/3641204.364121717:5(1081-1090)Online publication date: 2-May-2024
https://dl.acm.org/doi/10.14778/3641204.3641217
Cheng YChen PGuo CZhao KWen QYang BJensen C(2024)Weakly Guided Adaptation for Robust Time Series ForecastingProceedings of the VLDB Endowment10.14778/3636218.363623117:4(766-779)Online publication date: 5-Mar-2024
https://dl.acm.org/doi/10.14778/3636218.3636231
Liu KZhu YWang XJi HHuang C(2024)Spatio-Temporal Graph Attention Convolution Network for Traffic Flow ForecastingTransportation Research Record: Journal of the Transportation Research Board10.1177/036119812312252082678:9(136-149)Online publication date: 30-Jan-2024
https://doi.org/10.1177/03611981231225208
Yu CWang FShao ZQian TZhang ZWei WXu YBaeza-Yates RBonchi F(2024)GinAR: An End-To-End Multivariate Time Series Forecasting Model Suitable for Variable MissingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672055(3989-4000)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3672055
Gong JDing JMeng FChen GChen HZhao SLu HLi YBaeza-Yates RBonchi F(2024)A Population-to-individual Tuning Framework for Adapting Pretrained LM to On-device User Intent PredictionProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671984(896-907)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671984
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents