Hyperspherical Variational Co-embedding for Attributed Networks

Abstract

Network-based information has been widely explored and exploited in the information retrieval literature. Attributed networks, consisting of nodes, edges as well as attributes describing properties of nodes, are a basic type of network-based data, and are especially useful for many applications. Examples include user profiling in social networks and item recommendation in user-item purchase networks. Learning useful and expressive representations of entities in attributed networks can provide more effective building blocks to down-stream network-based tasks such as link prediction and attribute inference. Practically, input features of attributed networks are normalized as unit directional vectors. However, most network embedding techniques ignore the spherical nature of inputs and focus on learning representations in a Gaussian or Euclidean space, which, we hypothesize, might lead to less effective representations. To obtain more effective representations of attributed networks, we investigate the problem of mapping an attributed network with unit normalized directional features into a non-Gaussian and non-Euclidean space. Specifically, we propose a hyperspherical variational co-embedding for attributed networks (HCAN), which is based on generalized variational auto-encoders for heterogeneous data with multiple types of entities. HCAN jointly learns latent embeddings for both nodes and attributes in a unified hyperspherical space such that the affinities between nodes and attributes can be captured effectively. We argue that this is a crucial feature in many real-world applications of attributed networks. Previous Gaussian network embedding algorithms break the assumption of uninformative prior, which leads to unstable results and poor performance. In contrast, HCAN embeds nodes and attributes as von Mises-Fisher distributions, and allows one to capture the uncertainty of the inferred representations. Experimental results on eight datasets show that HCAN yields better performance in a number of applications compared with nine state-of-the-art baselines.

REFERENCES

1. [1] Aytekin Caglar, Ni Xingyang, Cricri Francesco, and Aksu Emre. 2018. Clustering and unsupervised anomaly detection with l 2 normalized deep auto-encoder representations. In Proceedings of the 2018 International Joint Conference on Neural Networks. IEEE, 1–6.
2. [2] Bojchevski Aleksandar and Günnemann Stephan. 2018. Deep gaussian embedding of graphs: Unsupervised inductive learning via ranking. In Proceedings of the International Conference on Learning Representations.
3. [3] Bowman Samuel R., Vilnis Luke, Vinyals Oriol, Dai Andrew M., Jozefowicz Rafal, and Bengio Samy. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. 10–21.
4. [4] Chakrabarti Deepayan, Funiak Stanislav, Chang Jonathan, and Macskassy Sofus A.. 2014. Joint inference of multiple label types in large networks. In Proceedings of the 31th International Conference on Machine Learning, Vol. 32. 874–882.
5. [5] Chami Ines, Abu-El-Haija Sami, Perozzi Bryan, Ré Christopher, and Murphy Kevin. 2020. Machine learning on graphs: A model and comprehensive taxonomy. arXiv preprint arXiv:2005.03675. 1–44.
6. [6] Chen Wanyu, Cai Fei, Chen Honghui, and Rijke Maarten de. 2019. Joint neural collaborative filtering for recommender systems. ACM Transactions on Information Systems 37, 4 (2019), 1–30.
7. [7] Cheng Yong. 2019. Semi-supervised learning for neural machine translation. In Proceedings of the Joint Training for Neural Machine Translation. Springer, 25–40.
8. [8] Cohen Taco, Geiger Mario, Köhler Jonas, and Welling Max. 2017. Convolutional networks for spherical signals. arXiv preprint arXiv:1709.04893. 1–5.
9. [9] Davidson Tim R., Falorsi Luca, Cao Nicola De, Kipf Thomas, and Tomczak Jakub M.. 2018. Hyperspherical variational auto-encoders. In Proceedings of the 34th Conference on Uncertainty in Artificial Intelligence (UAI’18). 856–865.
10. [10] Dhillon Inderjit S. and Sra Suvrit. 2003. Modeling Data using Directional Distributions. Technical Report TR-03-06. Department of Computer Sciences, The University of Texas at Austin.
11. [11] Santos Ludovic Dos, Piwowarski Benjamin, and Gallinari Patrick. 2016. Multilabel classification on heterogeneous graphs with gaussian embeddings. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 606–622.
12. [12] Fang Jinyuan, Liang Shangsong, Meng Zaiqiao, and Zhang Qiang. 2021. Gaussian process with graph convolutional kernel for relational learning. In Proceedings of the 27th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1–11.
13. [13] Fisher Nicholas I., Lewis Toby, and Embleton Brian J. J.. 1993. Statistical Analysis of Spherical Data. Cambridge University Press.
14. [14] Gao Hongchang and Huang Heng. 2018. Deep attributed network embedding. In Proceedings of the IJCAI, Vol. 18. 3364–3370.
15. [15] Gong Neil Zhenqiang, Talwalkar Ameet, Mackey Lester, Huang Ling, Shin Eui Chul Richard, Stefanov Emil, Shi Elaine Runting, and Song Dawn. 2014. Joint link prediction and attribute inference using a social-attribute network. ACM Transactions on Intelligent Systems and Technology 5, 2 (2014), 27.
16. [16] Gopal Siddharth and Yang Yiming. 2014. Von mises-fisher clustering models. In Proceedings of the International Conference on Machine Learning. 154–162.
17. [17] Gregor Karol, Danihelka Ivo, Graves Alex, Rezende Danilo Jimenez, and Wierstra Daan. 2015. Draw: A recurrent neural network for image generation. In Proceedings of the 32nd International Conference on Machine Learning, Vol. 37. 1462–1471.
18. [18] Grover Aditya and Leskovec Jure. 2016. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 855–864.
19. [19] Guu Kelvin, Hashimoto Tatsunori B., Oren Yonatan, and Liang Percy. 2018. Generating sentences by editing prototypes. Transactions of the Association of Computational Linguistics 6 (2018), 437–450.
20. [20] Hamilton Will, Ying Zhitao, and Leskovec Jure. 2017. Inductive representation learning on large graphs. In Proceedings of the Advances in Neural Information Processing Systems. 1024–1034.
21. [21] Hasnat Md, Bohné Julien, Milgram Jonathan, Gentric Stéphane, and Chen Liming. 2017. Von mises-fisher mixture model-based deep learning: Application to face verification. arXiv preprint arXiv:1706.04264. 1–16.
22. [22] He Shizhu, Liu Kang, Ji Guoliang, and Zhao Jun. 2015. Learning to represent knowledge graphs with gaussian embedding. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. ACM, 623–632.
23. [23] Hu Weihua, Fey Matthias, Zitnik Marinka, Dong Yuxiao, Ren Hongyu, Liu Bowen, Catasta Michele, and Leskovec Jure. 2020. Open graph benchmark: Datasets for machine learning on graphs. In Proceedings of the Advances in Neural Information Processing Systems.
24. [24] Huang Xiao, Li Jundong, and Hu Xia. 2017. Accelerated attributed network embedding. In Proceedings of the 2017 SIAM International Conference on Data Mining. SIAM, 633–641.
25. [25] Jiang Zhuxi, Zheng Yin, Tan Huachun, Tang Bangsheng, and Zhou Hanning. 2017. Variational deep embedding: An unsupervised and generative approach to clustering. In Proceedings of the 26th International Joint Conference on Artificial Intelligence. AAAI Press, 1965–1972.
26. [26] Kingma Diederik P. and Ba Jimmy. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations.
27. [27] Kingma Diederik P., Mohamed Shakir, Rezende Danilo Jimenez, and Welling Max. 2014. Semi-supervised learning with deep generative models. In Proceedings of the Advances in Neural Information Processing Systems. 3581–3589.
28. [28] Kingma Diederik P. and Welling Max. 2014. Auto-encoding variational bayes. In Proceedings of the International Conference on Learning Representations.
29. [29] Kipf Thomas N. and Welling Max. 2016. Variational graph auto-encoders. In Proceedings of the NIPS Workshop on Bayesian Deep Learning.
30. [30] Kipf Thomas N. and Welling Max. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of the International Conference on Learning Representations.
31. [31] Fond Timothy La and Neville Jennifer. 2010. Randomization tests for distinguishing social influence and homophily effects. In Proceedings of the 19th International Conference on World Wide Web. 601–610.
32. [32] Leskovec Jure and Mcauley Julian J.. 2012. Learning to discover social circles in ego networks. In Proceedings of the Advances in Neural Information Processing Systems. 539–547.
33. [33] Li Jundong, Dani Harsh, Hu Xia, Tang Jiliang, Chang Yi, and Liu Huan. 2017. Attributed network embedding for learning in a dynamic environment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. ACM, 387–396.
34. [34] Liang Dawen, Krishnan Rahul G., Hoffman Matthew D., and Jebara Tony. 2018. Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference. 689–698.
35. [35] Liang Shangsong, Luo Yupeng, and Meng Zaiqiao. 2021. Profiling users for question answering communities via flow-based constrained co-embedding model. In Proceedings of the ACM Transactions on Information Systems (2021), 1–38.
36. [36] Liang Shangsong, Yilmaz Emine, and Kanoulas Evangelos. 2016. Dynamic clustering of streaming short documents. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 995–1004.
37. [37] Liang Shangsong, Yilmaz Emine, and Kanoulas Evangelos. 2018. Collaboratively tracking interests for user clustering in streams of short texts. IEEE Transactions on Knowledge and Data Engineering 31, Issue 2 (2018), 257–272.
38. [38] Liang Shangsong, Zhang Xiangliang, Ren Zhaochun, and Kanoulas Evangelos. 2018. Dynamic embeddings for user profiling in Twitter. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1764–1773.
39. [39] Liao Yi, Lam Wai, Bing Lidong, and Shen Xin. 2018. Joint modeling of participant influence and latent topics for recommendation in event-based social networks. ACM Transactions on Information Systems 36, 3 (2018), 1–31.
40. [40] Maaten Laurens van der and Hinton Geoffrey. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 11 (2008), 2579–2605.
41. [41] Mardia Kantilal Varichand. 2014. Statistics of Directional Data. Academic press.
42. [42] Meng Zaiqiao, Liang Shangsong, Bao Hongyan, and Zhang Xiangliang. 2019. Co-embedding attributed networks. In Proceedings of the 12th ACM International Conference on Web Search and Data Mining. ACM, 393–401.
43. [43] Meng Zaiqiao, Liang Shangsong, Fang Jinyuan, and Xiao Teng. 2019. Semi-supervisedly co-embedding attributed networks. In Proceedings of the Advances in Neural Information Processing Systems. 6507–6516.
44. [44] Meng Zaiqiao, Liang Shangsong, Zhang Xiangliang, McCreadie Richard, and Ounis Iadh. 2020. Jointly learning representations of nodes and attributes for attributed networks. ACM Transactions on Information Systems 38, 2 (2020), 1–32.
45. [45] Mikolov Tomas, Sutskever Ilya, Chen Kai, Corrado Greg S., and Dean Jeff. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the Advances in Neural Information Processing Systems. 3111–3119.
46. [46] Naesseth Christian A., Ruiz Francisco J. R., Linderman Scott W., and Blei David M.. 2017. Reparameterization gradients through acceptance-rejection sampling algorithms. In Proceedings of the International Conference on Artificial Intelligence and Statistics, Vol. 54. 489–498.
47. [47] Paszke Adam, Gross Sam, Chintala Soumith, Chanan Gregory, Yang Edward, DeVito Zachary, Lin Zeming, Desmaison Alban, Antiga Luca, and Lerer Adam. 2017. Automatic differentiation in PyTorch. In Proceedings of the NIPS-W.
48. [48] Perozzi Bryan, Al-Rfou Rami, and Skiena Steven. 2014. Deepwalk: Online learning of social representations. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 701–710.
49. [49] Qu Meng, Bengio Yoshua, and Tang Jian. 2019. GMNN: Graph markov neural networks. In Proceedings of the International Conference on Machine Learning. PMLR, 5241–5250.
50. [50] Reinanda Ridho, Meij Edgar, and Rijke Maarten de. 2020. Knowledge graphs: An information retrieval perspective. Foundations and Trends in Information Retrieval 14, 4 (2020), 1–158.
51. [51] Reisinger Joseph, Waters Austin, Silverthorn Bryan, and Mooney Raymond J.. 2010. Spherical topic models. In Proceedings of the 27th International Conference on Machine Learning. 903–910.
52. [52] Rezende Danilo Jimenez, Mohamed Shakir, and Wierstra Daan. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the International Conference on Machine Learning, Vol. 32. 1278–1286.
53. [53] Sen Prithviraj, Namata Galileo, Bilgic Mustafa, Getoor Lise, Galligher Brian, and Eliassi-Rad Tina. 2008. Collective classification in network data. AI Magazine 29, 3 (2008), 93–93.
54. [54] Shchur Oleksandr, Mumme Maximilian, Bojchevski Aleksandar, and Günnemann Stephan. 2018. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868. 1–11.
55. [55] Sohn Kihyuk, Lee Honglak, and Yan Xinchen. 2015. Learning structured output representation using deep conditional generative models. In Proceedings of the Advances in Neural Information Processing Systems. 3483–3491.
56. [56] Tang Jian, Qu Meng, Wang Mingzhe, Zhang Ming, Yan Jun, and Mei Qiaozhu. 2015. Line: Large-scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1067–1077.
57. [57] Tang Lei and Liu Huan. 2009. Relational learning via latent social dimensions. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 817–826.
58. [58] Ulrich Gary. 1984. Computer generation of distributions on the M-Sphere. Journal of the Royal Statistical Society: Series C (Applied Statistics) 33, 2 (1984), 158–163.
59. [59] Veličković Petar, Cucurull Guillem, Casanova Arantxa, Romero Adriana, Lio Pietro, and Bengio Yoshua. 2017. Graph attention networks. In Proceedings of the International Conference on Learning Representations.
60. [60] Veličković Petar, Fedus William, Hamilton William L., Liò Pietro, Bengio Yoshua, and Hjelm R. Devon. 2018. Deep graph infomax. In Proceedings of the International Conference on Learning Representations.
61. [61] Wang Pengfei, Chen Hanxiong, Zhu Yadong, Shen Huawei, and Zhang Yongfeng. 2019. Unified collaborative filtering over graph embeddings. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 155–164.
62. [62] Wang Weiqing, Yin Hongzhi, Du Xingzhong, Hua Wen, Li Yongjun, and Nguyen Quoc Viet Hung. 2019. Online user representation learning across heterogeneous social networks. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 545–554.
63. [63] Wang Xiao, Cui Peng, Wang Jing, Pei Jian, Zhu Wenwu, and Yang Shiqiang. 2017. Community preserving network embedding. In Proceedings of the AAAI. 203–209.
64. [64] Wang Xiang, He Xiangnan, Wang Meng, Feng Fuli, and Chua Tat-Seng. 2019. Neural graph collaborative filtering. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. 165–174.
65. [65] Wu Libing, Quan Cong, Li Chenliang, Wang Qian, Zheng Bolong, and Luo Xiangyang. 2019. A context-aware user-item representation learning for item recommendation. ACM Transactions on Information Systems 37, 2 (2019), 1–29.
66. [66] Xiao Teng, Liang Shangsong, Shen Weizhou, and Meng Zaiqiao. 2019. Bayesian deep collaborative matrix factorization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 5474–5481.
67. [67] Xu Binbin, Pathak Sarthak, Fujii Hiromitsu, Yamashita Atsushi, and Asama Hajime. 2017. Spatio-temporal video completion in spherical image sequences. IEEE Robotics and Automation Letters 2, 4 (2017), 2032–2039.
68. [68] Xu Jiacheng and Durrett Greg. 2018. Spherical latent spaces for stable variational autoencoders. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 4503–4513.
69. [69] Yang Carl, Zhong Lin, Li Li-Jia, and Jie Luo. 2017. Bi-directional joint inference for user links and attributes on large social graphs. In Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 564–573.
70. [70] Yao Liang, Mao Chengsheng, and Luo Yuan. 2019. Graph convolutional networks for text classification. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 7370–7377.
71. [71] Yin Zhijun, Gupta Manish, Weninger Tim, and Han Jiawei. 2010. A unified framework for link recommendation using random walks. In Proceedings of the 2010 International Conference on Advances in Social Networks Analysis and Mining. IEEE, 152–159.
72. [72] Zhang Zhen, Yang Hongxia, Bu Jiajun, Zhou Sheng, Yu Pinggang, Zhang Jianwei, Ester Martin, and Wang Can. 2018. ANRL: Attributed network representation learning via deep neural networks. In Proceedings of the IJCAI. 3155–3161.
73. [73] Zhao Wayne Xin, Hou Yupeng, Chen Junhua, Zhu Jonathan J. H., Yin Eddy Jing, Su Hanting, and Wen Ji-Rong. 2020. Learning semantic representations from directed social links to tag microblog users at scale. ACM Transactions on Information Systems 38, 2 (2020), 1–30.

Index Terms

• Human-centered computing

  • Collaborative and social computing

    • Collaborative and social computing design and evaluation methods

      • Social network analysis

• Theory of computation

  • Theory and algorithms for application domains

    • Algorithmic game theory and mechanism design

      • Social networks