MARINA: An MLP-Attention Model for Multivariate Time-Series Analysis

ABSTRACT

The proliferation of real-time monitoring applications such as Artificial Intelligence for IT Operations (AIOps) and the Internet of Things (IoT) has led to the generation of a vast amount of time-series data. To extract the underlying value of the data, both the industry and the academia are in dire need of efficient and effective methods for time-series analysis. To this end, in this paper, we propose a Multi-layer perceptron (<u>M</u>LP)-<u>a</u>ttention based multivariate time-se<u>ri</u>es a<u>na</u>lysis model MARINA. MARINA is designed to simultaneously learn the temporal and spatial correlations among multivariate time-series. Also, the model is versatile in that it is suitable for major time-series analysis tasks such as forecasting and anomaly detection. Through extensive comparisons with the representative multivariate time-series forecasting and anomaly detection algorithms, MARINA is shown to achieve state-of-the-art (SOTA) performance in both forecasting and anomaly detection tasks.

References

1. Subutai Ahmad, Alexander Lavin, Scott Purdy, and Zuha Agha. 2017. Unsupervised real-time anomaly detection for streaming data. Neurocomputing 262 (Nov. 2017), 134--147.
2. Samet Akcay, Amir Atapour-Abarghouei, and Toby P Breckon. 2018. Ganomaly: Semi-supervised anomaly detection via adversarial training. In Asian Conference on Computer Vision. Springer, 622--637.
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv:1409.0473 [cs, stat] (May 2016). http://arxiv.org/abs/1409.0473 arXiv: 1409.0473.
4. Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2018. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv:1803.01271 (2018).
5. C. Chatfield. 1978. The Holt-Winters Forecasting Procedure. Journal of the Royal Statistical Society 27, 3 (1978), 264--279.
6. Xuanhao Chen, Liwei Deng, Feiteng Huang, Chengwei Zhang, Zongquan Zhang, Yan Zhao, and Kai Zheng. 2021. DAEMON: Unsupervised Anomaly Detection and Interpretation for Multivariate Time Series. In 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE.
7. Robert B. Cleveland, William S. Cleveland, Jean E. McRae, and Irma Terpenning. 1990. STL: A seasonal-trend decomposition. Journal of Official Statistics 6, 1 (1990), 3--73.
8. Yue Cui, Jiandong Xie, and Kai Zheng. 2021. Historical Inertia: An Ignored but Powerful Baseline for Long Sequence Time-series Forecasting. arXiv:2103.16349 (March 2021). http://arxiv.org/abs/2103.16349 arXiv: 2103.16349.
9. E. S. Gardner. 1985. Exponential smoothing: The state of the art. Journal of Forecasting 4, 1 (1985), 1--28.
10. Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2018. A Multimodal Anomaly Detector for Robot-Assisted Feeding Using An Lstm-Based Variational Autoencoder. IEEE Robotics and Automation Letters 3, 3 (July 2018), 1544--1551.
11. Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In AAAI 2019, Vol. 33. 922--929.
12. Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019. Attention Based Spatial-Temporal Graph Convolutional Networks for Traffic Flow Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence 33 (July 2019), 922--929.
13. E. J. Hannan and L. Kavalieris. 2010. REGRESSION, AUTOREGRESSION MODELS. Journal of Time 7, 1 (2010), 27--49.
14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, Las Vegas, NV, USA, 770--778.
15. Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell, and Tom Soderstrom. 2018. Detecting Spacecraft Anomalies Using LSTMs and Nonparametric Dynamic Thresholding. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, London United Kingdom, 387--395.
16. K. Kalpakis, D. Gada, and V. Puttagunta. 2002. Distance measures for effective clustering of ARIMA time-series. In Proceedings 2001 IEEE International Conference on Data Mining.
17. Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. arXiv:1312.6114 [cs, stat] (May 2014). http://arxiv.org/abs/1312.6114 arXiv: 1312.6114.
18. Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. arXiv:1609.02907 [cs, stat] (Feb. 2017). http://arxiv.org/abs/1609.02907 arXiv: 1609.02907.
19. Nikita Kitaev, çukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer. arXiv:2001.04451 [cs, stat] (Feb. 2020). http://arxiv.org/abs/2001. 04451 arXiv: 2001.04451.
20. Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, Ann Arbor MI USA, 95--104.
21. Nikolay Laptev, Saeed Amizadeh, and Ian Flint. 2015. Generic and Scalable Framework for Automated Time-series Anomaly Detection. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, Sydney NSW Australia, 1939--1947.
22. Dan Li, Dacheng Chen, Lei Shi, Baihong Jin, Jonathan Goh, and See-Kiong Ng. 2019. MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks. arXiv:1901.04997 [cs, stat] (Jan. 2019).
23. Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. 2020. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting. arXiv:1907.00235 [cs, stat] (Jan. 2020). http://arxiv.org/abs/1907.00235
24. Dapeng Liu, Youjian Zhao, Haowen Xu, Yongqian Sun, Dan Pei, Jiao Luo, Xiaowei Jing, and Mei Feng. 2015. Opprentice: Towards Practical and Automatic Anomaly Detection Through Machine Learning. In Proceedings of the 2015 Internet Measurement Conference. ACM, Tokyo Japan, 211--224.
25. Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. 2020. The M4 Competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting 36, 1 (Jan. 2020), 54--74.
26. Pankaj Malhotra, Anusha Ramakrishnan, Gaurangi Anand, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. 2016. LSTM-based Encoder-Decoder for Multisensor Anomaly Detection. arXiv:1607.00148 [cs, stat] (July 2016). http://arxiv.org/abs/1607.00148 arXiv: 1607.00148.
27. Boris N. Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. 2020. NBEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv:1905.10437 [cs, stat] (Feb. 2020). http://arxiv.org/abs/1905.10437 arXiv: 1905.10437.
28. F. Scarselli, M. Gori, Ah Chung Tsoi, M. Hagenbuchner, and G. Monfardini. 2009. The Graph Neural Network Model. IEEE Transactions on Neural Networks 20, 1 (Jan. 2009), 61--80.
29. Chao Song, Youfang Lin, Shengnan Guo, and HuaiyuWan. 2020. Spatial-Temporal Synchronous Graph Convolutional Networks: A New Framework for Spatial-Temporal Network Data Forecasting. Proceedings of the AAAI Conference on Artificial Intelligence 34 (April 2020), 914--921.
30. Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. 2016. Unsupervised Learning of Video Representations using LSTMs. arXiv:1502.04681 [cs] (Jan. 2016). http://arxiv.org/abs/1502.04681 arXiv: 1502.04681.
31. ya su, youjian zhao, Chenhao Niu, Rong Liu, Wei Sun, and Dan Pei. 2019. Robust Anomaly Detection for Multivariate Time Series through Stochastic Recurrent Neural Network. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, Anchorage Alaska USA, 2828--2837.
32. Sean Taylor and Benjamin Letham. 2018. Forecasting at scale. The American Statistician 72, 1 (2018), 37--74.
33. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. arXiv:1706.03762 [cs] (Dec. 2017). http://arxiv.org/abs/1706.03762 arXiv: 1706.03762.
34. Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. arXiv:1710.10903 [cs, stat] (Feb. 2018). http://arxiv.org/abs/1710.10903 arXiv: 1710.10903.
35. Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2022. Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting. arXiv:2106.13008 [cs.LG] (Jan. 2022). https://arxiv.org/abs/2106.13008 arXiv:2106.13008.
36. ZonghanWu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020. Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, Virtual Event CA USA, 753--763.
37. Haowen Xu, Wenxiao Chen, Nengwen Zhao, Zeyan Li, Jiahao Bu, Zhihan Li, Ying Liu, Youjian Zhao, Dan Pei, Yang Feng, et al. 2018. Unsupervised Anomaly Detection via Variational Auto-Encoder for Seasonal KPIs in Web Applications. In Proceedings of the 2018 WWW. 187--196.
38. X. Zhang, C. Huang, Y. Xu, and L. Xia. 2020. Spatial-Temporal Convolutional Graph Attention Networks for Citywide Traffic Flow Forecasting. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management.
39. Bin Zhou, Shenghua Liu, Bryan Hooi, Xueqi Cheng, and Jing Ye. 2019. BeatGAN: Anomalous Rhythm Detection using Adversarially Generated Time Series. In Proceedings of IJCAI. Macao, China, 4433--4439.
40. Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021. AAAI Press, online.
41. Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. 2018. Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection. (2018), 1--19.

Index Terms

• Computing methodologies

  • Machine learning