A Normalizing Flow-Based Co-Embedding Model for Attributed Networks

Abstract

Network embedding is a technique that aims at inferring the low-dimensional representations of nodes in a semantic space. In this article, we study the problem of inferring the low-dimensional representations of both nodes and attributes for attributed networks in the same semantic space such that the affinity between a node and an attribute can be effectively measured. Intuitively, this problem can be addressed by simply utilizing existing variational auto-encoder (VAE) based network embedding algorithms. However, the variational posterior distribution in previous VAE based network embedding algorithms is often assumed and restricted to be a mean-field Gaussian distribution or other simple distribution families, which results in poor inference of the embeddings. To alleviate the above defect, we propose a novel VAE-based co-embedding method for attributed network, F-CAN, where posterior distributions are flexible, complex, and scalable distributions constructed through the normalizing flow. We evaluate our proposed models on a number of network tasks with several benchmark datasets. Experimental results demonstrate that there are clear improvements in the qualities of embeddings generated by our model to the state-of-the-art attributed network embedding methods.

REFERENCES

1. [1] Abadi Martín, Barham Paul, Chen Jianmin, Chen Zhifeng, Davis Andy, Dean Jeffrey, Devin Matthieu, Ghemawat Sanjay, Irving Geoffrey, Isard Michael, Kudlur Manjunath, Levenberg Josh, Monga Rajat, Moore Sherry, Murray Derek Gordon, Steiner Benoit, Tucker Paul A., Vasudevan Vijay, Warden Pete, Wicke Martin, Yu Yuan, and Zheng Xiaoqiang. 2016. TensorFlow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation. 265–283.
2. [2] Adamic Lada A. and Adar Eytan. 2003. Friends and neighbors on the web. Social Networks 25, 3 (2003), 211–230.
3. [3] Ahmed Amr, Shervashidze Nino, Narayanamurthy Shravan M., Josifovski Vanja, and Smola Alexander J.. 2013. Distributed large-scale natural graph factorization. In Proceedings of the 22nd International World Wide Web Conference. 37–48.
4. [4] Aubin Benjamin, Loureiro Bruno, Maillard Antoine, Krzakala Florent, and Zdeborová Lenka. 2021. The spiked matrix model with generative priors. IEEE Transactions on Information Theory 67, 2 (2021), 1156–1181.
5. [5] Balunović Mislav, Ruoss Anian, and Vechev Martin. 2021. Fair normalizing flows. arXiv:2106.05937. Retrieved from https://arxiv.org/abs/2106.05937.
6. [6] Bojchevski Aleksandar and Günnemann Stephan. 2018. Deep Gaussian embedding of graphs: Unsupervised inductive learning via ranking. In Proceedings of the 6th International Conference on Learning Representations.
7. [7] Burda Yuri, Grosse Roger B., and Salakhutdinov Ruslan. 2016. Importance weighted autoencoders. In Proceedings of the 4th International Conference on Learning Representations.
8. [8] Cao Shaosheng, Lu Wei, and Xu Qiongkai. 2015. GraRep: Learning graph representations with global structural information. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. 891–900.
9. [9] Chakrabarti Deepayan, Funiak Stanislav, Chang Jonathan, and Macskassy Sofus A.. 2014. Joint inference of multiple label types in large networks. In Proceedings of the 31st International Conference on Machine Learning. 874–882.
10. [10] Davidson Tim R., Falorsi Luca, Cao Nicola De, Kipf Thomas, and Tomczak Jakub M.. 2018. Hyperspherical variational auto-encoders. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence. 856–865.
11. [11] Dinh Laurent, Krueger David, and Bengio Yoshua. 2015. NICE: Non-linear independent components estimation. In Proceedings of the 3rd International Conference on Learning Representations.
12. [12] Dinh Laurent, Sohl-Dickstein Jascha, and Bengio Samy. 2017. Density estimation using real NVP. In Proceedings of the 5th International Conference on Learning Representations.
13. [13] Gabrié Marylou. 2020. Mean-field inference methods for neural networks. Journal of Physics A: Mathematical and Theoretical 53, 22 (2020), 223002.
14. [14] Gabrié Marylou, Manoel Andre, Luneau Clément, Barbier Jean, Macris Nicolas, Krzakala Florent, and Zdeborová Lenka. 2018. Entropy and mutual information in models of deep neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 1826–1836.
15. [15] Gao Christina, Höche Stefan, Isaacson Joshua, Krause Claudius, and Schulz Holger. 2020. Event generation with normalizing flows. Physical Review D 101, 7 (2020), 076002.
16. [16] Gao Christina, Isaacson Joshua, and Krause Claudius. 2020. i-flow: High-dimensional integration and sampling with normalizing flows. Machine Learning: Science and Technology 1, 4 (2020), 045023.
17. [17] Gao Hongchang and Huang Heng. 2018. Deep attributed network embedding. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 3364–3370.
18. [18] Germain Mathieu, Gregor Karol, Murray Iain, and Larochelle Hugo. 2015. MADE: Masked autoencoder for distribution estimation. In Proceedings of the 32nd International Conference on Machine Learning. 881–889.
19. [19] Gong Neil Zhenqiang, Talwalkar Ameet, Mackey Lester, Huang Ling, Shin Eui Chul Richard, Stefanov Emil, Shi Elaine, and Song Dawn. 2014. Joint link prediction and attribute inference using a social-attribute network. ACM Transactions on Intelligent Systems and Technology 5, 2 (2014), 1–20.
20. [20] Gregor Karol, Danihelka Ivo, Graves Alex, Rezende Danilo, and Wierstra Daan. 2015. DRAW: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning. 1462–1471.
21. [21] Grover Aditya and Leskovec Jure. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 855–864.
22. [22] Hamilton William L., Ying Rex, and Leskovec Jure. 2017. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 1025–1035.
23. [23] Higgins Irina, Matthey Loic, Pal Arka, Burgess Christopher, Glorot Xavier, Botvinick Matthew, Mohamed Shakir, and Lerchner Alexander. 2017. beta-VAE: Learning basic visual concepts with a constrained variational framework. In Proceedings of the 5th International Conference on Learning Representations.
24. [24] Huang Xiao, Li Jundong, and Hu Xia. 2017. Accelerated attributed network embedding. In Proceedings of the 2017 SIAM International Conference on Data Mining. 633–641.
25. [25] Huang Xiao, Li Jundong, and Hu Xia. 2017. Label informed attributed network embedding. In Proceedings of the 10th ACM International Conference on Web Search and Data Mining. 731–739.
26. [26] Huang Xiao, Song Qingquan, Li Jundong, and Hu Xia. 2018. Exploring expert cognition for attributed network embedding. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 270–278.
27. [27] Kingma Diederik P. and Ba Jimmy. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.
28. [28] Kingma Diederik P. and Dhariwal Prafulla. 2018. Glow: Generative flow with invertible 1 \(\times\) 1 convolutions. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 10215–10224.
29. [29] Kingma Diederik P., Mohamed Shakir, Rezende Danilo Jimenez, and Welling Max. 2014. Semi-supervised learning with deep generative models. In Proceedings of the 27th International Conference on Neural Information Processing Systems. 3581–3589.
30. [30] Kingma Diederik P., Salimans Tim, Jozefowicz Rafal, Chen Xi, Sutskever Ilya, and Welling Max. 2016. Improving variational inference with inverse autoregressive flow. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 1–10.
31. [31] Kingma Diederik P. and Welling Max. 2014. Auto-encoding variational bayes. In Proceedings of the 2nd International Conference on Learning Representations.
32. [32] Kipf Thomas N. and Welling Max. 2016. Variational graph auto-encoders. stat 1050 (2016), 21 pages.
33. [33] Kipf Thomas N. and Welling Max. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the 5th International Conference on Learning Representations.
34. [34] Le Quoc and Mikolov Tomas. 2014. Distributed representations of sentences and documents. In Proceedings of the 31st International Conference on Machine Learning. 1188–1196.
35. [35] Li Jundong, Cheng Kewei, Wu Liang, and Liu Huan. 2018. Streaming link prediction on dynamic attributed networks. In Proceedings of the 11th ACM International Conference on Web Search and Data Mining. 369–377.
36. [36] Li Jundong, Dani Harsh, Hu Xia, Tang Jiliang, Chang Yi, and Liu Huan. 2017. Attributed network embedding for learning in a dynamic environment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 387–396.
37. [37] Li Ye, Sha Chaofeng, Huang Xin, and Zhang Yanchun. 2018. Community detection in attributed graphs: An embedding approach. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence. 338–345.
38. [38] Liang Jiongqian, Jacobs Peter, Sun Jiankai, and Parthasarathy Srinivasan. 2018. Semi-supervised embedding in attributed networks with outliers. In Proceedings of the 2018 SIAM International Conference on Data Mining. 153–161.
39. [39] Liang Shangsong. 2018. Dynamic user profiling for streams of short texts. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32.
40. [40] Liang Shangsong. 2019. Collaborative, dynamic and diversified user profiling. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence. 4269–4276.
41. [41] Liang Shangsong, Yilmaz Emine, and Kanoulas Evangelos. 2018. Collaboratively tracking interests for user clustering in streams of short texts. IEEE Transactions on Knowledge and Data Engineering 31, 2 (2018), 257–272.
42. [42] Liang Shangsong, Zhang Xiangliang, Ren Zhaochun, and Kanoulas Evangelos. 2018. Dynamic embeddings for user profiling in Twitter. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1764–1773.
43. [43] Liao Siyuan, Liang Shangsong, Meng Zaiqiao, and Zhang Qiang. 2021. Learning dynamic embeddings for temporal knowledge graphs. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 535–543.
44. [44] Liu Jenny, Kumar Aviral, Ba Jimmy, Kiros Jamie, and Swersky Kevin. 2019. Graph normalizing flows. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. 13556–13566.
45. [45] Liu Zhining, Zhou Dawei, and He Jingrui. 2019. Towards explainable representation of time-evolving graphs via spatial-temporal graph attention networks. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 2137–2140.
46. [46] Liu Zhining, Zhou Dawei, Zhu Yada, Gu Jinjie, and He Jingrui. 2020. Towards fine-grained temporal network representation via time-reinforced random walk. In Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34, 4973–4980.
47. [47] Luo Yupeng, Liang Shangsong, and Meng Zaiqiao. 2019. Constrained co-embedding model for user profiling in question answering communities. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. 439–448.
48. [48] McAuley Julian J. and Leskovec Jure. 2012. Learning to discover social circles in ego networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems.548–556.
49. [49] Meng Zaiqiao, Liang Shangsong, Bao Hongyan, and Zhang Xiangliang. 2019. Co-embedding attributed networks. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining. 393–401.
50. [50] Meng Zaiqiao, Liang Shangsong, Fang Jinyuan, and Xiao Teng. 2019. Semi-supervisedly co-embedding attributed networks. In Proceedings of the 33rd International Conference on Neural Information Processing Systems. 4743–4751.
51. [51] Meng Zaiqiao, Liang Shangsong, Zhang Xiangliang, McCreadie Richard, and Ounis Iadh. 2020. Jointly learning representations of nodes and attributes for attributed networks. ACM Transactions on Information Systems 38, 2 (2020), 1–32.
52. [52] Mikolov Tomas, Sutskever Ilya, Chen Kai, Corrado Greg, and Dean Jeffrey. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Advances in Neural Information processing Systems. 3111–3119.
53. [53] Mnih Andriy and Gregor Karol. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on International Conference on Machine Learning. 1791–1799.
54. [54] Ou Mingdong, Cui Peng, Pei Jian, Zhang Ziwei, and Zhu Wenwu. 2016. Asymmetric transitivity preserving graph embedding. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 1105–1114.
55. [55] Pandey Gaurav and Dukkipati Ambedkar. 2016. Variational methods for conditional multimodal learning: Generating human faces from attributes. arXiv:1603.01801. Retrieved from https://arxiv.org/abs/1603.01801.
56. [56] Papamakarios George, Murray Iain, and Pavlakou Theo. 2017. Masked autoregressive flow for density estimation. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 2338–2347.
57. [57] Perozzi Bryan, Al-Rfou Rami, and Skiena Steven. 2014. DeepWalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 701–710.
58. [58] Pu Yunchen, Gan Zhe, Henao Ricardo, Yuan Xin, Li Chunyuan, Stevens Andrew, and Carin Lawrence. 2016. Variational autoencoder for deep learning of images, labels and captions. In Proceedings of the 30th International Conference on Neural Information Processing Systems. Vol. 29, 2352–2360.
59. [59] Rezende Danilo Jimenez and Mohamed Shakir. 2015. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference on Machine Learning. PMLR, 1530–1538.
60. [60] Rezende Danilo Jimenez, Mohamed Shakir, and Wierstra Daan. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on International Conference on Machine Learning. 1278–1286.
61. [61] Sen Prithviraj, Namata Galileo, Bilgic Mustafa, Getoor Lise, Galligher Brian, and Eliassi-Rad Tina. 2008. Collective classification in network data. AI Magazine 29, 3 (2008), 93.
62. [62] Simonovsky Martin and Komodakis Nikos. 2018. GraphVAE: Towards generation of small graphs using variational autoencoders. In Proceedings of the International Conference on Artificial Neural Networks. Springer, 412–422.
63. [63] Sohn Kihyuk, Lee Honglak, and Yan Xinchen. 2015. Learning structured output representation using deep conditional generative models. In Proceedings of the 28th International Conference on Neural Information Processing Systems. 3483–3491.
64. [64] Sønderby Casper Kaae, Raiko Tapani, Maaløe Lars, Sønderby Søren Kaae, and Winther Ole. 2016. Ladder variational autoencoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems. 3738–3746.
65. [65] Sun Heli, He Fang, Huang Jianbin, Sun Yizhou, Li Yang, Wang Chenyu, He Liang, Sun Zhongbin, and Jia Xiaolin. 2020. Network embedding for community detection in attributed networks. ACM Transactions on Knowledge Discovery from Data 14, 3 (2020), 1–25.
66. [66] Tang Jian, Qu Meng, Wang Mingzhe, Zhang Ming, Yan Jun, and Mei Qiaozhu. 2015. LINE: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web. 1067–1077.
67. [67] Tang Lei and Liu Huan. 2009. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 817–826.
68. [68] Turner Richard, Sahani M.. 2011. Two problems with variational expectation maximisation for time-series models. In Bayesian Time Series Models. Barber D., Cemgil T., and Chiappa S. (Eds.), Cambridge University Press, 109–130.
69. [69] Wang Hao, Chen Enhong, Liu Qi, Xu Tong, Du Dongfang, Su Wen, and Zhang Xiaopeng. 2018. A united approach to learning sparse attributed network embedding. In Proceedings of the 2018 IEEE International Conference on Data Mining. 557–566.
70. [70] Wang Suhang, Aggarwal Charu, Tang Jiliang, and Liu Huan. 2017. Attributed signed network embedding. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 137–146.
71. [71] Wang Xiao, Cui Peng, Wang Jing, Pei Jian, Zhu Wenwu, and Yang Shiqiang. 2017. Community preserving network embedding. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.
72. [72] Wu Hao, Köhler Jonas, and Noé Frank. 2020. Stochastic normalizing flows. arXiv:2002.06707. Retrieved from https://arxiv.org/abs/2002.06707.
73. [73] Wu Wei, Li Bin, Chen Ling, and Zhang Chengqi. 2018. Efficient attributed network embedding via recursive randomized hashing. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 2861–2867.
74. [74] Yang Carl, Zhong Lin, Li Li-Jia, and Jie Luo. 2017. Bi-directional joint inference for user links and attributes on large social graphs. In Proceedings of the 26th International Conference on World Wide Web. 564–573.
75. [75] Yang Hong, Pan Shirui, Zhang Peng, Chen Ling, Lian Defu, and Zhang Chengqi. 2018. Binarized attributed network embedding. In Proceedings of the 2018 IEEE International Conference on Data Mining. 1476–1481.
76. [76] Zanfir Andrei, Bazavan Eduard Gabriel, Xu Hongyi, Freeman Bill, Sukthankar Rahul, and Sminchisescu Cristian. 2020. Weakly supervised 3d human pose and shape reconstruction with normalizing flows. In Proceedings of the European Conference on Computer Vision. Springer, 465–481.
77. [77] Zang Chengxi and Wang Fei. 2020. MoFlow: An invertible flow model for generating molecular graphs. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 617–626.
78. [78] Zhang Zhen, Yang Hongxia, Bu Jiajun, Zhou Sheng, Yu Pinggang, Zhang Jianwei, Ester Martin, and Wang Can. 2018. ANRL: Attributed network representation learning via deep neural networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 3155–3161.
79. [79] Zhou Dawei, He Jingrui, Yang Hongxia, and Fan Wei. 2018. Sparc: Self-paced network representation for few-shot rare category characterization. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2807–2816.
80. [80] Zhu Dingyuan, Cui Peng, Wang Daixin, and Zhu Wenwu. 2018. Deep variational network embedding in Wasserstein space. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2827–2836.

Index Terms

• Computing methodologies

  • Machine learning

    • Machine learning approaches

      • Neural networks

• Information systems

  • Information systems applications

    • Data mining