skip to main content
research-article
Artifacts Available / v1.1

Zebra: When Temporal Graph Neural Networks Meet Temporal Personalized PageRank

Published:20 April 2023Publication History
Skip Abstract Section

Abstract

Temporal graph neural networks (T-GNNs) are state-of-the-art methods for learning representations over dynamic graphs. Despite the superior performance, T-GNNs still suffer from high computational complexity caused by the tedious recursive temporal message passing scheme, which hinders their applicability to large dynamic graphs. To address the problem, we build the theoretical connection between the temporal message passing scheme adopted by T-GNNs and the temporal random walk process on dynamic graphs. Our theoretical analysis indicates that it would be possible to select a few influential temporal neighbors to compute a target node's representation without compromising the predictive performance. Based on this finding, we propose to utilize T-PPR, a parameterized metric for estimating the influence score of nodes on evolving graphs. We further develop an efficient single-scan algorithm to answer the top-k T-PPR query with rigorous approximation guarantees. Finally, we present Zebra, a scalable framework that accelerates the computation of T-GNN by directly aggregating the features of the most prominent temporal neighbors returned by the top-k T-PPR query. Extensive experiments have validated that Zebra can be up to two orders of magnitude faster than the state-of-the-art T-GNNs while attaining better performance.

References

  1. [2023]. AskUbuntu. http://snap.stanford.edu/data/sx-askubuntu.html.Google ScholarGoogle Scholar
  2. [2023]. SuperUser. http://snap.stanford.edu/data/sx-superuser.html.Google ScholarGoogle Scholar
  3. [2023]. The technical report. https://github.com/LuckyLYM/Zebra/blob/main/technical_report.pdf.Google ScholarGoogle Scholar
  4. [2023]. Wiki-talk. http://snap.stanford.edu/data/wiki-talk-temporal.html.Google ScholarGoogle Scholar
  5. Reid Andersen, Christian Borgs, Jennifer T. Chayes, John E. Hopcroft, Vahab S. Mirrokni, and Shang-Hua Teng. 2008. Local Computation of PageRank Contributions. Internet Mathematics 5, 1 (2008), 23--45.Google ScholarGoogle ScholarCross RefCross Ref
  6. Reid Andersen, Fan R. K. Chung, and Kevin J. Lang. 2006. Local Graph Partitioning using PageRank Vectors. In FOCS. IEEE Computer Society, 475--486.Google ScholarGoogle Scholar
  7. Bahman Bahmani, Abdur Chowdhury, and Ashish Goel. 2010. Fast Incremental and Personalized PageRank. PVLDB 4, 3 (2010), 173--184.Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Aleksandar Bojchevski, Johannes Klicpera, Bryan Perozzi, Amol Kapoor, Martin Blais, Benedek Rózemberczki, Michal Lukasik, and Stephan Günnemann. 2020. Scaling Graph Neural Networks with Approximate PageRank. In KDD. ACM, 2464--2473.Google ScholarGoogle Scholar
  9. Ben Chamberlain, James Rowbottom, Maria I. Gorinova, Michael M. Bronstein, Stefan Webb, and Emanuele Rossi. 2021. GRAND: Graph Neural Diffusion. In ICML (Proceedings of Machine Learning Research), Vol. 139. PMLR, 1407--1418.Google ScholarGoogle Scholar
  10. Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. 2020. Measuring and Relieving the Over-Smoothing Problem for Graph Neural Networks from the Topological View. In AAAI. AAAI Press, 3438--3445.Google ScholarGoogle Scholar
  11. Yasuhiro Fujiwara, Makoto Nakatsuji, Makoto Onizuka, and Masaru Kitsuregawa. 2012. Fast and Exact Top-k Search for Random Walk with Restart. PVLDB 5, 5 (2012), 442--453.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Yasuhiro Fujiwara, Makoto Nakatsuji, Hiroaki Shiokawa, Takeshi Mishima, and Makoto Onizuka. 2013. Efficient ad-hoc search for personalized PageRank. In SIGMOD. ACM, 445--456.Google ScholarGoogle Scholar
  13. Manuel Gomez-Rodriguez, David Balduzzi, and Bernhard Schölkopf. 2011. Uncovering the Temporal Dynamics of Diffusion Networks. In ICML. Omnipress, 561--568.Google ScholarGoogle Scholar
  14. Manuel Gomez-Rodriguez, Jure Leskovec, and Andreas Krause. 2010. Inferring networks of diffusion and influence. In KDD. ACM, 1019--1028.Google ScholarGoogle Scholar
  15. Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. 2020. dyngraph2vec: Capturing network dynamics using dynamic graph representation learning. Knowledge Based System 187 (2020).Google ScholarGoogle Scholar
  16. Ehsan Hajiramezanali, Arman Hasanzadeh, Krishna R. Narayanan, Nick Duffield, Mingyuan Zhou, and Xiaoning Qian. 2019. Variational Graph Recurrent Neural Networks. In NeurIPS. 10700--10710.Google ScholarGoogle Scholar
  17. William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In NIPS. 1024--1034.Google ScholarGoogle Scholar
  18. Thomas N. Kipf and Max Welling. 2016. Variational Graph Auto-Encoders. CoRR abs/1611.07308 (2016).Google ScholarGoogle Scholar
  19. Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. 2019. Predict then Propagate: Graph Neural Networks meet Personalized PageRank. In ICLR. OpenReview.net.Google ScholarGoogle Scholar
  20. Johannes Klicpera, Stefan Weißenberger, and Stephan Günnemann. 2019. Diffusion Improves Graph Learning. In NeurIPS. 13333--13345.Google ScholarGoogle Scholar
  21. Pang Wei Koh and Percy Liang. 2017. Understanding Black-box Predictions via Influence Functions. In ICML (Proceedings of Machine Learning Research), Vol. 70. PMLR, 1885--1894.Google ScholarGoogle Scholar
  22. Srijan Kumar, Xikun Zhang, and Jure Leskovec. 2019. Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks. In SIGKDD. ACM, 1269--1278.Google ScholarGoogle Scholar
  23. Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015. Numba: a LLVM-based Python JIT compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, LLVM 2015, Austin, Texas, USA, November 15, 2015, Hal Finkel (Ed.). ACM, 7:1--7:6.Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Guohao Li, Matthias Müller, Ali K. Thabet, and Bernard Ghanem. 2019. Deep-GCNs: Can GCNs Go As Deep As CNNs?. In ICCV. IEEE, 9266--9275.Google ScholarGoogle Scholar
  25. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. [n.d.]. A Diversity-Promoting Objective Function for Neural Conversation Models. In HLT-NAACL. 110--119.Google ScholarGoogle Scholar
  26. Dandan Lin, Raymond Chi-Wing Wong, Min Xie, and Victor Junqiu Wei. 2020. Index-Free Approach with Theoretical Guarantee for Efficient Random Walk with Restart Query. In ICDE. IEEE, 913--924.Google ScholarGoogle Scholar
  27. Siyang Liu, Sahand Sabour, Yinhe Zheng, Pei Ke, Xiaoyan Zhu, and Minlie Huang. [n.d.]. Rethinking and Refining the Distinct Metric. In ACL. 762--770.Google ScholarGoogle Scholar
  28. Peter Lofgren, Siddhartha Banerjee, and Ashish Goel. 2016. Personalized PageRank Estimation and Search: A Bidirectional Approach. In WSDM. ACM, 163--172.Google ScholarGoogle Scholar
  29. Dingheng Mo and Siqiang Luo. 2021. Agenda: Robust Personalized PageRanks in Evolving Graphs. In CIKM, Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, 1315--1324.Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Giang Hoang Nguyen, John Boaz Lee, Ryan A. Rossi, Nesreen K. Ahmed, Eunyee Koh, and Sungchul Kim. 2018. Continuous-Time Dynamic Network Embeddings. In WWW. ACM, 969--976.Google ScholarGoogle Scholar
  31. Kenta Oono and Taiji Suzuki. 2020. Graph Neural Networks Exponentially Lose Expressive Power for Node Classification. In ICLR. OpenReview.net.Google ScholarGoogle Scholar
  32. Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The PageRank Citation Ranking: Bringing Order to the Web. Technical Report 1999-66. Stanford InfoLab.Google ScholarGoogle Scholar
  33. Aldo Pareja, Giacomo Domeniconi, Jie Chen, Tengfei Ma, Toyotaro Suzumura, Hiroki Kanezashi, Tim Kaler, Tao B. Schardl, and Charles E. Leiserson. 2020. EvolveGCN: Evolving Graph Convolutional Networks for Dynamic Graphs. In AAAI. 5363--5370.Google ScholarGoogle Scholar
  34. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS. 8024--8035.Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Allan Pinkus. 1999. Approximation theory of the MLP model in neural networks. Acta numerica 8 (1999), 143--195.Google ScholarGoogle Scholar
  36. Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael M. Bronstein. 2020. Temporal Graph Networks for Deep Learning on Dynamic Graphs. CoRR abs/2006.10637 (2020).Google ScholarGoogle Scholar
  37. Emanuele Rossi, Fabrizio Frasca, Ben Chamberlain, Davide Eynard, Michael M. Bronstein, and Federico Monti. 2020. SIGN: Scalable Inception Graph Neural Networks. CoRR abs/2004.11198 (2020). arXiv:2004.11198Google ScholarGoogle Scholar
  38. Polina Rozenshtein and Aristides Gionis. 2016. Temporal PageRank. In ECML (Lecture Notes in Computer Science), Vol. 9852. Springer, 674--689.Google ScholarGoogle Scholar
  39. Omer Sagi and Lior Rokach. 2018. Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8, 4 (2018).Google ScholarGoogle Scholar
  40. Jieming Shi, Renchi Yang, Tianyuan Jin, Xiaokui Xiao, and Yin Yang. 2019. Realtime Top-k Personalized PageRank over Large Graphs on GPUs. PVLDB 13, 1 (2019), 15--28.Google ScholarGoogle ScholarDigital LibraryDigital Library
  41. Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. 2019. DyRep: Learning Representations over Dynamic Graphs. In ICLR.Google ScholarGoogle Scholar
  42. Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. In ICLR.Google ScholarGoogle Scholar
  43. Hanzhi Wang, Mingguo He, Zhewei Wei, Sibo Wang, Ye Yuan, Xiaoyong Du, and Ji-Rong Wen. 2021. Approximate Graph Propagation. In KDD. ACM, 1686--1696.Google ScholarGoogle Scholar
  44. Hongwei Wang and Jure Leskovec. 2020. Unifying Graph Convolutional Neural Networks and Label Propagation. CoRR abs/2002.06755 (2020).Google ScholarGoogle Scholar
  45. Hanzhi Wang, Zhewei Wei, Junhao Gan, Ye Yuan, Xiaoyong Du, and Ji-Rong Wen. 2022. Edge-based Local Push for Personalized PageRank. CoRR abs/2203.07937 (2022).Google ScholarGoogle Scholar
  46. Sibo Wang, Youze Tang, Xiaokui Xiao, Yin Yang, and Zengxiang Li. 2016. HubPPR: Effective Indexing for Approximate Personalized PageRank. PVLDB 10, 3 (2016), 205--216.Google ScholarGoogle ScholarDigital LibraryDigital Library
  47. Sibo Wang, Renchi Yang, Runhui Wang, Xiaokui Xiao, Zhewei Wei, Wenqing Lin, Yin Yang, and Nan Tang. 2019. Efficient Algorithms for Approximate Single-Source Personalized PageRank Queries. ACM Transactions on Database Systems 44, 4 (2019), 18:1--18:37.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. Sibo Wang, Renchi Yang, Xiaokui Xiao, Zhewei Wei, and Yin Yang. 2017. FORA: Simple and Effective Approximate Single-Source Personalized PageRank. In KDD. ACM, 505--514.Google ScholarGoogle ScholarDigital LibraryDigital Library
  49. Xuhong Wang, Ding Lyu, Mengjian Li, Yang Xia, Qi Yang, Xinwen Wang, Xinguang Wang, Ping Cui, Yupu Yang, Bowen Sun, and Zhenyu Guo. 2021. APAN: Asynchronous Propagation Attention Network for Real-time Temporal Graph Embedding. In SIGMOD. ACM, 2628--2638.Google ScholarGoogle Scholar
  50. Yanbang Wang, Yen-Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. 2021. Inductive Representation Learning in Temporal Networks via Causal Anonymous Walks. In ICLR.Google ScholarGoogle Scholar
  51. Zhewei Wei, Xiaodong He, Xiaokui Xiao, Sibo Wang, Shuo Shang, and Ji-Rong Wen. 2018. TopPPR: Top-k Personalized PageRank Queries with Precision Guarantees on Large Graphs. In SIGMOD, Gautam Das, Christopher M. Jermaine, and Philip A. Bernstein (Eds.). ACM, 441--456.Google ScholarGoogle Scholar
  52. Hao Wu, Junhao Gan, Zhewei Wei, and Rui Zhang. 2021. Unifying the Global and Local Approaches: An Efficient Power Iteration with Forward Push. In SIGMOD. ACM, 1996--2008.Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. Yubao Wu, Ruoming Jin, and Xiang Zhang. 2014. Fast and unified local search for random walk based k-nearest-neighbor query in large graphs. In SIGMOD. ACM, 1139--1150.Google ScholarGoogle Scholar
  54. Da Xu, Chuanwei Ruan, Evren Körpeoglu, Sushant Kumar, and Kannan Achan. 2020. Inductive representation learning on temporal graphs. In ICLR.Google ScholarGoogle Scholar
  55. Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018. Representation Learning on Graphs with Jumping Knowledge Networks. In ICML (Proceedings of Machine Learning Research), Vol. 80. PMLR, 5449--5458.Google ScholarGoogle Scholar
  56. Minji Yoon, Woojeong Jin, and U Kang. 2018. Fast and Accurate Random Walk with Restart on Dynamic Graphs with Guarantees. In WWW. ACM, 409--418.Google ScholarGoogle Scholar
  57. Hongyang Zhang, Peter Lofgren, and Ashish Goel. 2016. Approximate Personalized PageRank on Dynamic Graphs. In KDD. ACM, 1315--1324.Google ScholarGoogle Scholar
  58. Wentao Zhang, Yu Shen, Zheyu Lin, Yang Li, Xiaosen Li, Wen Ouyang, Yangyu Tao, Zhi Yang, and Bin Cui. 2021. GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passing. CoRR abs/2104.09880 (2021).Google ScholarGoogle Scholar
  59. Wentao Zhang, Zhi Yang, Yexin Wang, Yu Shen, Yang Li, Liang Wang, and Bin Cui. 2021. Grain: Improving Data Efficiency of Graph Neural Networks via Diversified Influence Maximization. PVLDB 14, 11 (2021), 2473--2482.Google ScholarGoogle ScholarDigital LibraryDigital Library
  60. Hongkuan Zhou, Da Zheng, Israt Nisa, Vasileios Ioannidis, Xiang Song, and George Karypis. 2022. TGL: A General Framework for Temporal GNN Training on Billion-Scale Graphs. PVLDB 15, 8 (apr 2022), 1572--1580.Google ScholarGoogle Scholar
  61. Zhi-Hua Zhou. 2012. Ensemble Methods: Foundations and Algorithms. Vol. 14.Google ScholarGoogle ScholarCross RefCross Ref

Recommendations

Comments

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Sign in

Full Access

PDF Format

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader
About Cookies On This Site

We use cookies to ensure that we give you the best experience on our website.

Learn more

Got it!