METRO: a generic graph neural network framework for multivariate time series forecasting

Abstract

Multivariate time series forecasting has been drawing increasing attention due to its prevalent applications. It has been commonly assumed that leveraging latent dependencies between pairs of variables can enhance prediction accuracy. However, most existing methods suffer from static variable relevance modeling and ignorance of correlation between temporal scales, thereby failing to fully retain the dynamic and periodic interdependencies among variables, which are vital for long- and short-term forecasting. In this paper, we propose METRO, a generic framework with multi-scale temporal graphs neural networks, which models the dynamic and cross-scale variable correlations simultaneously. By representing the multivariate time series as a series of temporal graphs, both intra- and inter-step correlations can be well preserved via message-passing and node embedding update. To enable information propagation across temporal scales, we design a novel sampling strategy to align specific steps between higher and lower scales and fuse the cross-scale information efficiently. Moreover, we provide a modular interpretation of existing GNN-based time series forecasting works as specific instances under our framework. Extensive experiments conducted on four benchmark datasets demonstrate the effectiveness and efficiency of our approach. METRO has been successfully deployed onto the time series analytics platform of Huawei Cloud, where a one-month online test demonstrated that up to 20% relative improvement over state-of-the-art models w.r.t. RSE can be achieved.

References

1. George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. 2015. Time series analysis: forecasting and control. John Wiley & Sons.
2. Peter J Brockwell, Richard A Davis, and Matthew V Calder. 2002. Introduction to time series and forecasting. Vol. 2. Springer.
3. Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP'14.
4. Yue Cui, Jiandong Xie, and Kai Zheng. 2021. Historical Inertia: A Neglected but Powerful Baseline for Long Sequence Time-series Forecasting. In CIKM'21.
5. Yue Cui, Chen Zhu, Guanyu Ye, Ziwei Wang, and Zheng Kai. 2021. Into the Unobservables: A Multi-range Encoder-decoder Framework for COVID-19 Prediction. In CIKM'21.
6. Shib Sankar Dasgupta, Swayambhu Nath Ray, and Partha Talukdar. 2018. Hyte: Hyperplane-based temporally aware knowledge graph embedding. In Proceedings of the 2018 conference on empirical methods in natural language processing. 2001--2011.
7. Roger Frigola, Fredrik Lindsten, Thomas B Schön, and Carl Edward Rasmussen. 2013. Bayesian Inference and Learning in Gaussian Process State-Space Models with Particle MCMC. In NIPS'13. 3156--3164.
8. Hardik Goel, Igor Melnyk, and Arindam Banerjee. 2017. R2n2: Residual recurrent neural networks for multivariate time series forecasting. arXiv preprint arXiv:1709.03159 (2017).
9. Rishab Goel, Seyed Mehran Kazemi, Marcus Brubaker, and Pascal Poupart. 2020. Diachronic embedding for temporal knowledge graph completion. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 3988--3995.
10. İsmail Güneş, Şule Gündüz-Öğüdücü, and Zehra Çataltepe. 2016. Link prediction using time series of neighborhood-based node similarity scores. Data Mining and Knowledge Discovery 30, 1 (2016), 147--180.
11. Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019. Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting. (2019), 922--929.
12. Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. 2020. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting (2020).
13. Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. 2012. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 14, 8 (2012), 2.
14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780.
15. Siteng Huang, Donglin Wang, Xuehan Wu, and Ao Tang. 2019. DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting. In CIKM'19. 2129--2132.
16. Zan Huang and Dennis KJ Lin. 2009. The time-series link prediction problem with applications in communication surveillance. INFORMS Journal on Computing 21, 2 (2009), 286--303.
17. Nahla Mohamed Ahmed Ibrahim and Ling Chen. 2015. Link prediction in dynamic social networks by integrating different types of information. Applied Intelligence 42, 4 (2015), 738--750.
18. Dino Ienco and Roberto Interdonato. 2020. Deep Multivariate Time Series Embedding Clustering via Attentive-Gated Autoencoder. In Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 318--329.
19. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR'15. 1395--1402.
20. Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In ICLR'17.
21. Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. In SIGIR'18. 95--104.
22. Maosen Li, Siheng Chen, Yangheng Zhao, Ya Zhang, Yanfeng Wang, and Qi Tian. 2020. Dynamic Multiscale Graph Neural Networks for 3D Skeleton Based Human Motion Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 214--223.
23. David Liben-Nowell and Jon Kleinberg. 2007. The link-prediction problem for social networks. Journal of the American society for information science and technology 58, 7 (2007), 1019--1031.
24. Yao Ma, Ziyi Guo, Zhaocun Ren, Jiliang Tang, and Dawei Yin. 2020. Streaming graph neural networks. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 719--728.
25. Giang Hoang Nguyen, John Boaz Lee, Ryan A Rossi, Nesreen K Ahmed, Eunyee Koh, and Sungchul Kim. 2018. Continuous-time dynamic network embeddings. In Companion Proceedings of the The Web Conference 2018. 969--976.
26. Giang H Nguyen, John Boaz Lee, Ryan A Rossi, Nesreen K Ahmed, Eunyee Koh, and Sungchul Kim. 2018. Dynamic network embeddings: From random walks to temporal random walks. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 1085--1092.
27. Herbert Robbins and Sutton Monro. 1951. A stochastic approximation method. The annals of mathematical statistics (1951), 400--407.
28. Stephen Roberts, Michael Osborne, Mark Ebden, Steven Reece, Neale Gibson, and Suzanne Aigrain. 2013. Gaussian Processes for time-series modelling. Philosophical Transactions Mathematical Physical & Engineering Sciences (2013).
29. Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael Bronstein. 2020. Temporal Graph Networks for Deep Learning on Dynamic Graphs. arXiv preprint arXiv:2006.10637 (2020).
30. Rajat Sen, Hsiang-Fu Yu, and Inderjit S Dhillon. 2019. Think globally, act locally: A deep neural network approach to high-dimensional time series forecasting. In NIPS'19. 4837--4846.
31. Umang Sharan and Jennifer Neville. 2008. Temporal-relational classifiers for prediction in evolving domains. In 2008 Eighth IEEE International Conference on Data Mining. IEEE, 540--549.
32. Qiquan Shi, Jiaming Yin, Jiajun Cai, Andrzej Cichocki, Tatsuya Yokota, Lei Chen, Mingxuan Yuan, and Jia Zeng. 2020. Block Hankel Tensor ARIMA for Multiple Short Time Series Forecasting.. In AAAI'20. 5758--5766.
33. Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. 2019. Temporal pattern attention for multivariate time series forecasting. In Machine Learning, Vol. 108. 1421--1441.
34. Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu C Aggarwal, Prasenjit Mitra, and Suhang Wang. 2020. Joint Modeling of Local and Global Temporal Dynamics for Multivariate Time Series Forecasting with Missing Values.. In AAAI'20. 5956--5963.
35. Anh Tong and Jaesik Choi. 2019. Discovering latent covariance structures for multiple time series. In ICML'19. PMLR, 6285--6294.
36. Rakshit Trivedi, Hanjun Dai, Yichen Wang, and Le Song. 2017. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In international conference on machine learning. PMLR, 3462--3471.
37. Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. 2019. Dyrep: Learning representations over dynamic graphs. In International conference on learning representations.
38. Vladimir Vapnik, Steven E Golowich, and Alex J Smola. 1997. Support vector method for function approximation, regression estimation and signal processing. In NIPS'97. 281--287.
39. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In NIPS'17.
40. Andrew A Weiss. 1984. Systematic sampling and temporal aggregation in time series models. Journal of Econometrics 26, 3 (1984), 271--281.
41. Billy M Williams and Lester A Hoel. 2003. Modeling and forecasting vehicular traffic flow as a seasonal ARIMA process: Theoretical basis and empirical results. Journal of transportation engineering 129, 6 (2003), 664--672.
42. Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. In KDD'20.
43. Yu Xin, Zhi-Qiang Xie, and Jing Yang. 2016. An adaptive random walk sampling method on dynamic community detection. Expert Systems with Applications 58 (2016), 10--19.
44. Chenjin Xu, Mojtaba Nayyeri, Fouad Alkhoury, Hamed Yazdi, and Jens Lehmann. 2020. Temporal knowledge graph completion based on time series gaussian embedding. In International Semantic Web Conference. Springer, 654--671.
45. Dongkuan Xu, Wei Cheng, Bo Zong, Dongjin Song, Jingchao Ni, Wenchao Yu, Yanchi Liu, Haifeng Chen, and Xiang Zhang. 2020. Tensorized LSTM with Adaptive Shared Memory for Learning Trends in Multivariate Time Series.. In AAAI'20. 1395--1402.
46. Haoyan Xu, Yida Huang, Ziheng Duan, Jie Feng, and Pengyu Song. 2020. Multi-variate Time Series Forecasting Based on Causal Inference with Transfer Entropy and Graph Neural Network. arXiv preprint arXiv:2005.01185 (2020).
47. Huaxiu Yao, Xianfeng Tang, Hua Wei, Guanjie Zheng, and Zhenhui Li. 2019. Revisiting spatial-temporal similarity: A deep learning framework for traffic prediction. In AAAI'19, Vol. 33. 5668--5675.
48. Hsiang-Fu Yu, Nikhil Rao, and Inderjit S Dhillon. 2016. Temporal regularized matrix factorization for high-dimensional time series prediction. In NIPS'16. 847--855.
49. Wenchao Yu, Wei Cheng, Charu C Aggarwal, Haifeng Chen, and Wei Wang. 2017. Link Prediction with Spatial and Temporal Consistency in Dynamic Networks.. In IJCAI. 3343--3349.
50. Wenchao Yu, Wei Cheng, Charu C Aggarwal, Kai Zhang, Haifeng Chen, and Wei Wang. 2018. Netwalk: A flexible deep embedding approach for anomaly detection in dynamic networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2672--2681.
51. G Peter Zhang. 2003. Time series forecasting using a hybrid ARIMA and neural network model. Neurocomputing 50 (2003), 159--175.
52. Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021. AAAI Press, online.

Index Terms

• Computing methodologies

  • Machine learning

    • Machine learning approaches

      • Neural networks

• Information systems

  • Information systems applications

• Mathematics of computing

• Theory of computation

  • Theory and algorithms for application domains

    • Machine learning theory