Toward Effective Semi-supervised Node Classification with Hybrid Curriculum Pseudo-labeling

Semi-supervised node classification is a crucial challenge in relational data mining and has attracted increasing interest in research on graph neural networks (GNNs). However, previous approaches merely utilize labeled nodes to supervise the overall optimization, but fail to sufficiently explore the information of their underlying label distribution. Even worse, they often overlook the robustness of models, which may cause instability of network outputs to random perturbations. To address the aforementioned shortcomings, we develop a novel framework termed Hybrid Curriculum Pseudo-Labeling (HCPL) for efficient semi-supervised node classification. Technically, HCPL iteratively annotates unlabeled nodes by training a GNN model on the labeled samples and any previously pseudo-labeled samples, and repeatedly conducts this process. To improve the model robustness, we introduce a hybrid pseudo-labeling strategy that incorporates both prediction confidence and uncertainty under random perturbations, therefore mitigating the influence of erroneous pseudo-labels. Finally, we leverage the idea of curriculum learning to start from annotating easy samples, and gradually explore hard samples as the iteration grows. Extensive experiments on a number of benchmarks demonstrate that our HCPL beats various state-of-the-art baselines in diverse settings.


INTRODUCTION
With the increasing popularity of cloud computing technologies and the Internet of Things, as well as the expansion of social media, structured data is rising at an unprecedented rate.A graph is an effective and powerful tool for representing a large number of relational data across various domains including biology, social networks, information science, and so on [25].Thus, the exploration of graph-structured data is very critical and necessary.In recent years, graph neural networks (GNNs) have been proposed and shown incredible performance in studying graphstructured data.Typically, GNNs are capable of combining vertex attributes and graph topology information to learn vertex representations for a variety of downstream graph-based tasks, including node classification [29,31,54,79], graph classification [27,35], graph clustering [26,70], link prediction [7,73], and traffic forecasting [23,37].Here in this article, we study semi-supervised node classification, which aims to forecast the categories of unlabeled nodes using a limited number of labeled nodes.
Indeed, various GNN algorithms for semi-supervised node classification have been developed [21,29,52], the bulk of which rely on the construction of diverse neighborhood propagation strategies for learning effective node representations.The most prominent technique is graph convolutional networks (GCNs) [29], which aggregate node representations of their local neighbors iteratively.Following GCNs, a number of graph convolutions have been developed sequentially using a variety of message passing algorithms.For example, Graph Attention Network (GAT) [52] integrates the attention mechanism into the message passing process that enables feature information to be passed adaptively.By eliminating nonlinearities and compressing weight matrices between successive layers, Simple Graph Convolution (SGC) [60] reduces the computational cost of a GCN.Hamilton et al. [21] propose to sample node neighborhoods for higher efficiency.Graph Isomorphism Network (GIN) [66] further improves the expressive capability of GCNs and is capable of capturing different graph structural information.GNNs have also been applied to various applications such as multi-view learning [69] and recommendation [40,67].
Despite the remarkable performance in semi-supervised node classification, existing approaches go through two critical constraints that may impair model performance.On the one hand, these methods typically leverage the unlabeled nodes when propagating their attributes during the message passing process using GNNs, while ignoring the information of underlying label distribution.This problem may lead to easy underfitting, particularly in the absence of adequate annotated labels, thus limiting the performance of the network [54].On the other hand, they usually pay less attention to the robustness of the model.Existing GNNs often have predefined attributes and neighborhood propagation patterns, which leads to each node being extremely reliant on its initial features and neighbors.When networks are attacked by noise in real-world applications [75], they may output unstable predictions, leading to considerable performance deterioration.
Numerous semi-supervised learning algorithms have been comprehensively investigated in recent years for making full use of the unlabeled datum [9,30,41,50,51].Pseudo-labeling [30] is one of the most classic methods in this field.It needs a predictor to iteratively output the categories of unlabeled samples and involve well-classified examples in the training dataset.On this basis, further works usually encourage the classifier to make predictions with a small entropy on unlabeled samples [20].Pseudo-labeling techniques have a lot of downstream applications in various tasks.For example, FixMatch [49] combines pseudo-labeling and consistency regularization to address the problems of label scarcity for image classification.Cross Pseudo Supervision (CPS) [9] introduces pseudo-labeling techniques into semantic segmentation problems.However, pseudolabeling techniques are predominantly studied in the visual domains but have not been well applied to effectively solving node classification problems on graphs yet.Therefore, it is promising to explore unlabeled nodes on the graph with semi-supervised techniques.
Towards Effective Semi-supervised Node Classification with HCPL 82:3 Toward this end, this article develops a simple but effective approach called Hybrid Curriculum Pseudo-Labeling (HCPL) for semi-supervised node classification.The core of our idea is to sufficiently explore the unlabeled data through a pseudo-labeling strategy.Specifically, we iteratively annotate unlabeled nodes by training a model on the labeled samples and any previously pseudo-labeled samples, and repeat the process in a self-training way.Moreover, we not only involve a perturbed GNN predictor with the adaptive decoupling of the representation transformation and neighborhood propagation, but also introduce a hybrid pseudo-labeling strategy to increase the robustness under noise attack.We take both prediction confidence and uncertainty into account while dealing with noise, alleviating the impact of potential erroneous pseudo-labels.Note that curriculum learning [4] is a training strategy that trains a learning model from easier data to harder data.Inspired by this, we utilize the idea of curriculum learning to annotate easy samples with high confidence and robustness and then annotate hard samples with less confidence and robustness.This strategy learns the model in a meaningful order and helps the model free from error accumulation.In this way, our HCPL can sufficiently explore the unlabeled data through a pseudo-labeling strategy.Experimental results on several popular benchmark datasets demonstrate our HCPL outperforms a wide range of state-of-the-art approaches.To sum up, the contributions of this work are as follows: -We develop a novel approach named HCPL for semi-supervised node classification, which leverages curriculum learning to produce confident pseudo-labels to make full use of the abundant unlabeled data while existing works usually do not explore semantic information in unlabeled data.-We study the model robustness and introduce a novel hybrid pseudo-labeling strategy that takes into consideration both prediction confidence and prediction uncertainty to produce accurate pseudo-labels.-Extensive experiments on six graph datasets show that HCPL achieves remarkable performance compared with a variety of state-of-the-arts in different settings.

GNNs
Recent years have witnessed increasing attention in research to apply deep learning methods to graph-structured data.GNNshave come into the spotlight due to their superior capability for graph representation learning [32,34,47,55,64] with wide applications such as fake new detection [24,68].Pioneering efforts use spectral-based approaches for localized and effective graph convolution.GCN simplifies GNN into spatial-based models [29], which utilizes the adjacent matrix and increases the effectiveness of GNNs.These spatial-based approaches are typically based on a neighborhood aggregation mechanism that updates node representation via the aggregation of information from its neighbors.Following that, several GCN variants have been developed subsequently, including GAT [52], SGC [60], and GIN [66].GAT incorporates the attention mechanism to assess the importance of different neighbors on the center node and uses the attention scores as feature aggregation weights.Inspired by the Weisfeiler-Lehman algorithm [46], GIN [66] improves the expressive capability of GCNs via capturing different graph structures.However, the majority of GNN-based methods train the model using the cross entropy on labeled nodes to optimize the model, which neglect the abundant unlabeled nodes.In contrast, our research builds on the strength of GNNs and focuses on enhancing semi-supervised node classification via curriculum learning in a pseudo-labeling way.GNNs can also be applied to various settings such as PU learning [61], open-world learning [62], unsupervised domain adaptation [63], and crossmodal retrieval [39].For example, LSDAN [61] incorporates the attention mechanism with GNN to 82:4 X. Luo et al.
model node significance from both short and long terms.OpenWGL [62] utilizes a variation graph autoencoder to explore unseen nodes in the test set.GCLN [63] models both attraction and repulsion forces for consistency learning within a single graph and across graphs.DAGNN [39] adopts multi-hop GNN to investigate the relationship between labels for effective cross-modal retrieval.

Semi-supervised Learning
Semi-supervised learning (SSL) has recently drawn a lot of interest and achieved a lot of success in a variety of fields.SSL is capable of reducing the need for labeled data by using a vast volume of unlabeled data.Because unlabeled data can be quickly obtained with minimal human effort, the performance of models may be improved at a low cost using SSL.Two mainstream methodologies of SSL are self-training and consistency regularization.The pioneer semi-supervised learning works are based on self-training (so-called pseudo-labeling) [8,30,48,51], which uses the class predictions of models as pseudo-labels to train unlabeled data in a supervised way.Specifically, unlabeled samples are iteratively added to the training data by annotating them with a weak model trained with labeled data.Another line of SSL is based on recent breakthroughs in consistency learning [5,49], which encourages the network to make consistent predictions when it comes to noise perturbation on unlabeled samples.Semi-supervised learning techniques have been extensively utilized in a variety of fields such as computer vision and knowledge mining [1,9,13,43,49].
Inspired by recent advances in visual domains, our proposed HCPL effectively combines curriculum learning and semi-supervised learning, and develops a novel pseudo-labeling strategy for effective semi-supervised node classification on graphs.Our work is also related to teaching-to-learn and learning-to-teach (TLLT) [15], which also adopt an easy-to-hard curriculum strategy for graph-based semi-supervised learning.TLLT aims to perform the message propagation based on the teacher module and choose the following samples from the learner module in an alternative manner.ML-TLLT [18] introduces this framework to solve the problem of multi-labeled learning, which studies every possible label for the unlabeled samples along with the label dependencies.SMMCL [19] incorporates curriculum learning to learn the procedure of label propagation on graphs.MMCL [17] leverages curriculum learning to acquire the difficulty of accurately classifying each unlabeled sample for semi-supervised image classification.Gong et al. [16] adopt TLLT into the saliency detection, which successfully identifies the salient objects in images with high propagation quality.

Graph-based Semi-supervised Learning
Semi-supervised node classification is the most fundamental problem in graph data mining [14,29,33,56,59], which has various applications in social analysis [42], bioinformatics [65], image annotation [76], text generation [58], and noise cleaning [74].For example, ASFS [72] explores the pairwise relationships in the latent and then utilizes the graph structure to guide semantics learning in a semi-supervised setting.This work utilizes a classic feature selection framework while ours adopts GNNs for semi-supervised node classification.For semi-supervised node classification, only a few annotation nodes are available in the graph to predict the labels of the remaining nodes.Traditional methods of solving this problem are typically based on graph Laplacian regularizations [3,36,77].For example, Belkin et al. [3] propose to exploit the geometry of the marginal distribution and give a new form of regularization.Recently, GNNs have emerged as a powerful approach to learn from the graph.However, existing GNN methods usually focus on developing effective message passing patterns [29,60], but neglect to sufficiently exploit the information of unlabeled nodes.Our article, by contrast, tackles this issue through effective pseudo-labeling for better semi-supervised node classification.We believe our method can be extended to tackle various graph-related semi-supervised tasks [12].

METHOD
To begin with, we introduce the problem definition and present our approach HCPL for semisupervised node classification on graphs.Previous methods usually neglect the label information contained in unlabeled nodes as well as the robustness of the model.To tackle the issues, our approach is based on the exploration of unlabeled data via pseudo-labeling.Namely, we annotate unlabeled nodes by training a model on labeled samples and any previously pseudo-labeled data and then repeating this procedure in a self-training fashion.Specifically, we first propose a perturbed GNN predictor and present a hybrid pseudo-labeling technique, which takes both prediction confidence and uncertainty into account under noise attack.Finally, an optimization pipeline embraced with curriculum learning is used to dynamically and automatically select pseudo-labels.
The overall framework can be illustrated in Figure 1.

Problem Formulation
A graph is represented in a form of G = (V, E), in which V denotes a set of N nodes and E ⊆ V ×V denotes a set of edges in the graph.x i ∈ R F denotes the attribute feature of node v i , where F is the dimension number of the node attributes.In addition, each node v i is associated with a label vector, i.e., y i ∈ {0, 1} C where C is the number of label categories.In our case, M (M < N ) nodes in V L are annotated with their labels, while the labels of the other N − M nodes are unknown.
Our aim is to predict the unobserved labels for unlabeled nodes in V U in a graph.Take the social network CORA as an example.Each node v i corresponds to a research paper, and two research papers (i.e., v i and v j ) are linked if the paper v i is cited by the paper v j .There are seven classes of topics, i.e., reinforcement learning, neural networks, case-based research, genetic algorithms, probabilistic methods, rule learning, and theory.We need to classify all papers without labels into their associated classes.

Perturbed GNN Predictor
Here, we present a perturbed GNN as the backbone of our HCPL.GNNs have been frequently utilized to collect node attributive information as well as topological information on graphs using deep neural networks.We begin with the introduction of the popular message passing procedure.
In formulation, the embedding of node v i ∈ V at the layer k is represented as h (k )  i .The message 82:6 X. Luo et al.
passing procedure in GNNs usually involves two steps: (i) Aggregation step, which collects the semantic information from the neighborhood of node v i at the previous layer k − 1; and (ii) Combination step, which merges the node embedding of v i at the previous layer with the obtained neighbor embedding at the current layer.To summarize, where N (v i ) denotes the neighborhood of v i , and AGG (k ) θ and COM (k )  θ represent the aggregation and combination operators at the layer k, respectively.After performing neighborhood aggregation K times, the embedding vectors at all layers are condensed into a single embedding vector: where SUM θ represents the summarization operator.Widely used mean aggregator, LSTM aggregator, and pooling aggregator [21] can also be utilized to generate informative and structure-aware representations for various downstream tasks.
In our implementation, we begin with a Multi-Layer Perception (MLP) to process the initial attributes.Formally, we have which will be concatenated into feature matrix Z ∈ R |V |×d and d is the embedding dimension.Then, given the adjacent matrix A ∈ R |V |×|V | , we use the decoupled symmetrical normalization propagation to generate node embeddings at every layer.
where A = D − 1 2 A D − 1 2 and A = A + I .Finally, we use an attention mechanism [2,33] to aggregate embeddings at all layers.Formally, the summarized node representation h i for node v i is written as where h k i is the representation of the i-th node at the layer k, σ (•) denotes an activation function, and W is a trainable matrix to acquire the weight.
Nonetheless, neighborhood propagation strategies are typically predefined in GNNs, leaving each node largely reliant on its attributes and neighbors.The neural network could be misled during graph convolution methods due to noise attacks on node properties and connection patterns.Here, we utilize two popular graph augmentation strategies [71] to make it easier to generate disturb-invariant node representations.
-Attribute Masking: We choose several vertices and mask a portion of their attributes afterward.The prior behind this strategy is that masking part of vertices will not change the semantics of the node much.It serves as the dropout strategy in the deep neural network.-Edge Deletion: Certain edges are randomly dropped out of the graph based on an i.i.d uniform distribution.The strategy corresponds to the prior that the node semantics should be robust to the random attacks of edge connectivity patterns.
Towards Effective Semi-supervised Node Classification with HCPL 82:7 The augmented version of G is denoted as G.After our GNN, we feed the representation h i of each node into a two-layer MLP classifier to produce the prediction vector p i ∈ R C .Formally, where θ is the parameter of the GNN predictor.

Hybrid Pseudo-Label Selection
In our framework, we first optimize the model using labeled data and then employ the trained model to output the label distribution for unlabeled data.Furthermore, we seek to collect accurate pseudo-labels and add these unlabeled data into the training set.Previous methods [30] usually utilize the confidence scores for pseudo-labeling, which could generate biased and overconfident samples, and therefore result in error accumulation.Therefore, we propose a hybrid pseudo-label selection strategy based on both the confidence and uncertainty of the prediction, which will be elaborated as follows.
Confidence-based Selection.Intuitively, based on the label distribution, we seek to select hard samples with high confidence.Formally, let p c i denote the probability of the i-th example belonging to the class c, and the pseudo-label can be obtained as follows: where γ is a fixed threshold.Note that the use of hard labels is associated with entropy minimization [20], where the model prediction is enforced to be of low entropy on unlabeled samples.Uncertainty-based Selection.However, the accuracy of pseudo-labels is still far from satisfactory, since it does not take the robustness of the network into consideration.From a different perspective, the prediction is unreliable if the predicted distribution is unstable to random attack [43].At this inspirit, we re-run the perturbed GNN predictor for W times where W is the running number of the predictor, and calculate the uncertainty of pseudo-labels using the formulation of standard deviation.Let p c i denote the list of W predictions, and we have where η is another threshold and sd(•) denotes the standard error of the W predictions.In this way, we consider the robustness of our model by selecting pseudo-labels invariant to random attacks, and thus our model is more likely to produce correct pseudo-labels.Finally, we combine the advantages of the two strategies by taking the intersection of selected pseudo-labels.Note that, since we forward the GNN predictor for W times, we use the mean of the prediction to replace the single output in Equation (7).In formulation, where mean(p c i ) denotes the mean of the W predictions.In this way, the pseudo-labels with both high confidence and robustness will be selected, which greatly improves the accuracy of pseudolabels.In summary, the motivation of our strategy is to evaluate the difficulty of classifying each sample accurately, which can guide sequential curriculum learning.Here, our hybrid pseudo-label selection strategy is introduced based on both the confidence and uncertainty of the prediction.On the one hand, we select samples with high maximal probabilities, for which the model has high confidence about the prediction.On the other hand, we re-run the perturbed GNN predictor and evaluate the variance of the predictions, which can reflect the uncertainty of the prediction.These uncertainty scores can help us to evaluate the difficulty from different views.Finally, we take the intersection of results from our hybrid strategy to detect reliable samples under both rules.Further  ablation studies also validate the effectiveness of our hybrid strategy and we believe our method can be utilized in more semi-supervised settings such as cross-modal retrieval.

Optimization Pipeline with Curriculum Learning
In our framework, we first train the GNN predictor using labeled data.Specifically, we employ the standard cross-entropy loss to train labeled nodes on the augmented graphs.Formally, Then, following the principle of self-training, we output the prediction for unlabeled data for W times, and then select reliable pseudo-labels based on Equation (9).These unlabeled nodes and their pseudo-labels are added to the labeled subset.
Note that a fixed threshold in selection is not optimal.For example, a large threshold γ may lead to too few pseudo-labels while a small value may bring in too many inaccurate pseudo-labels otherwise.As a result, we involve in a novel pipeline by adopting curriculum learning, resulting in dynamic thresholds in the selection for multiple iterations.To be specific, we gradually select more unlabeled samples from easy to difficult by increasing η and decreasing γ .In our implementation, we use the percentile of scores to decide the thresholds.Assume the total number of iterations is T and arдmax c ∈C {mean(p c i )} = c i .For the t-th iteration, the thresholds are adjusted with γ t = Percentile mean p , 100 − 100/T * t , where Percentile (S, m) denotes the values of the m-th percentile of set S. Then, we select pseudolabels with dynamic thresholds as follows: Note that at T -th iteration, all the unlabeled nodes will be exhausted, i.e., annotated in selftraining.An example is illustrated in Figure 2. Through curriculum learning [8], we start from easy samples with high confidence and low uncertainty, and gradually explore hard unlabeled samples.We are also involved in two strategies.First, after each iteration, we restore the labeled set and re-annotate every node in the unlabeled set.This enables pseudo-annotated nodes to enter or leave the updated set.Second, we train the GNN predictor from scratch, i.e., reinitialize the parameters in the GNN predictor after each iteration instead of popular fine-tuning.These two strategies can discourage concept drift or confirmation bias introduced at the early stage of selftraining to be accumulated, improving the performance of our proposed HCPL.The whole pipeline of the optimization process is illustrated in Algorithm 1.

EXPERIMENTS
In this part, by conducting extensive experiments on six real-world datasets to show the effectiveness of our HCPL, we highlight the following results: Forward the networks for W times to get the mean together with the standard deviation of the predictions for all nodes in V U , i.e., mean(p c i ) and sd (p c i ) 4: Calculate the percentiles r 1 and r 2 for thresholds γ t and η t , respectively, with Equation ( Restore current labeled set ṼL using V L 6: end if 10: Train GNN model using updated ṼL from scratch 11: end for 12: end for -HCPL significantly outperforms all competing baselines that are compared to all experimental settings.-Ablation studies demonstrate the efficiency of the different components of HCPL.
-The performance of our methods is stable to main hyper-parameters in proper ranges.
-Our HCPL is robust to random attack compared with baselines.

Experimental Setup
Datasets.Our HCPL is accessed on six widely used benchmark node classification datasets including three paper citation datasets [6,44], i.e., Cora, CiteSeer, and PubMed, two purchasing graph datasets [45], i.e., Amazon Computers and Amazon Photo, and one co-author network dataset [45], i.e., Coauthor CS.In three paper citation datasets, nodes denote publication and edges denote citation links.The purpose is to classify these nodes into different areas.Both purchasing graph datasets are collected from Amazon, where nodes represent goods and edges are constructed when two goods are often bought at the same time.CoauthorCS is a co-author network dataset where nodes denote authors and edges indicate co-author relationships.The statistics of these datasets are summarized in Table 1.
We utilize the same splits in the previous work [54] to construct train/validation/test datasets for three citation datasets, while for the other three datasets, we randomly choose 30 nodes from each category as labeled training data, 30 nodes as validation data, and other nodes as the test data.For a fair comparison, we adopt the same dataset splits on all datasets for all baseline methods.Compared Methods.To evaluate the effectiveness of our developed HCPL, we compare it with the following state-of-the-art baseline models for semi-supervised node classification as follows.
-Chebyshev [10]: It is a formulation of CNNs that leverages the idea of spectral graph theory to devise fast localized convolutional filters suitable for graph data.-GCN [29]: It is a classic semi-supervised GNN model based on the spectral theory that generates node representations via aggregating information from neighbors.-GAT [52]: It is a GNN model that improves GCN by incorporating the attention mechanism to assign different weights to each neighboring node of a node.-SGC [60]: It is a fast algorithm that lowers the unnecessary computational cost of GCN via removing the nonlinearity between layers and compressing the weight matrix.
-DGI [53]: It is an unsupervised approach for learning node representations, which focuses on the mutual information between node-level representations and their associated graph-level representations.
-GMI [38]: It presents a new method for measuring the similarity degree between input graphs and hidden node embeddings, generalizing the concept of mutual information computations to the graph domain.
-MVGRL [22]: It introduces a self-supervised approach, which learns node-level and graphlevel representations by maximizing mutual information between representations encoded from different topological views of graphs.-GRACE [78]: It is a novel framework based on contrastive learning for unsupervised graph representation learning via a hybrid scheme for generating graph views on both topology and feature levels.-CG 3 [54]: It is a novel GCN-based semi-supervised learning algorithm that enriches the label information via leveraging node similarities and structural knowledge from two different perspectives.-AM-GCN [57]: It fuses multi-view information from topological structures and features using the attention mechanism.
Parameter Settings.We implement all the compared methods using PyTorch 1.8.0 and Pytorch Geometric 1.7.2, which are capable of smoothly training GNNs for a range of applications connected to graph-structured data.Extensive experiments are performed on an NVIDIA GeForce GTX 1080 Ti.For simplicity, we adopt a two-layer GCN [29] as the GCN backbone as default and include a model variant HCPL-A that utilizes GAT [52] as the backbone.The dimension number of hidden embedding is set to 256 for all datasets and the number of iterations is set to 20.These two hyper-parameters will be discussed in Section 4.4.Adam [28] is employed during optimization due to its effectiveness.We set the learning rate to 0.01 and it decays with the rate 0.0005.
For all experiments, we present the mean accuracy with standard deviations from five runs.The validation dataset is utilized to tune all hyper-parameters, and the test dataset can provide the final results.For the parameters in the adopted baselines, we refer to their original papers and utilize their tuning strategies for the best performance.

Experimental Results
Table 2 displays the compared results on six datasets.From the table, the following observations can be obtained: -GCN-based algorithms (i.e., GCN, GAT, and SGC) overall perform better than the traditional method (i.e., Chebyshev), which shows that the superior representation-learning ability of GCN helps to enhance the performance for semi-supervised node classification.[29] 81.3 ± 0.4 71.5 ± 0.2 78.8 ± 0.6 77.7 ± 0.7 88.1 ± 0.8 91.6 ± 0.7 GAT [52] 82.7 ± 0.1 70.7 ± 0.4 78.5 ± 0.2 79.5 ± 0.2 88.0 ± 0.6 91.2 ± 0.5 SGC [60] 77.7 ± 0.0 72.6 ± 0.0 76.4 ± 0.0 74.8 ± 0.1 87.9 ± 0.1 90.2 ± 0.2 DGI [53] 80.9 ± 0.3 71.4 ± 0.2 76.3 ± 1.1 77.7 ± 0.8 85.3 ± 0.9 90.6 ± 0.5 GMI [38] 81.6 ± 0.4 71.9 ± 0.5 81.8 ± 0.4 78.9 ± 0.1 84.9 ± 0.0 90.7 ± 0.0 MVGRL [22] 81.3 ± 0.4 71.9 ± 0.1 79.3 ± 0.1 79.5 ± 0.8 88.1 ± 0.2 91.7 ± 0.1 AM-GCN [57] 81.0 ± 0.3 72.8 ± 0.4 OOM 80.9 ± 0.7 91.3 ± 0.2 OOM GRACE [78] 82.8 ± 0. -The methods (i.e., DGI, GMI, MVGRL, GRACE, CG 3 , and HCPL) that explore the representations or label distribution of unlabeled data perform better than other methods, showing that utilizing additional unlabeled datum by unsupervised or semi-supervised learning is an important complement to supervised learning, enhancing model performance.-In all of the datasets, our approach produces the best results.In particular, on the large-scale datasets Amazon Computers and Amazon Photo, our HCPL outperforms the best baseline CG 3 by 2.1% and 2.6%, respectively, validating the efficiency of our HCPL.We claim this improvement can be attributed to two reasons: (i) Our hybrid pseudo-labeling strategy incorporates both prediction confidence and uncertainty to generate accurate pseudo-labels for unlabeled nodes.(ii) Our curriculum learning pipeline gradually explores unlabeled samples to avoid overconfident annotations.-We have conducted one-sample paired t-tests to justify that the improvements with the best baseline are statistically significant with p-value < 0.05 on all the datasets.However, the variance of our HCPL is a little larger than that of baselines on several datasets.A potential reason is that in some cases, wrong pseudo-labels could make the performance a little unstable when studying unlabeled nodes.In practice, we suggest running multiple times and selecting the best model based on validation datasets.We calculate the classification accuracy on all test nodes.-The performance improvement compared with the best baseline (CG 3 ) is limited in Cora.
The potential reason could be the high homophily ratio of Cora and thus low risk of biased pseudo-labeling, which makes curriculum learning less important.In practice, we can measure the risk of biased pseudo-labeling by active learning and then design the algorithm accordingly.
Moreover, we can validate that GAT can still benefit from our hybrid curriculum pseudo-labeling by comparing GAT and HCPL-A.Moreover, our HCPL-A can perform better than all the baselines in most cases, which validates our superiority again.Of note, our HCPL is trained in an iterative way.Hence, we access the accuracy of our HCPL at each iteration to observe whether the performance will improve as the number of iterations grows under curriculum learning.The results of the three datasets are shown in Figure 3.We can observe that the performance increases in most cases after each iteration, which validates that our proposed HCPL benefits from pseudo-labeling and curriculum learning.Further, we experiment in the cases where the labeled samples are changed to access the performance of the HCPL in handling different supervision.We choose a proportion of labeled samples for model training in each run following [54].We first choose the Cora dataset as an example where the label rates are varying in 0.5%, 1%, 2%, 3%, 5%, 10%, 20%, and 50%.The result is summarized in Table 3. Again, we can see that our HCPL consistently beats other baselines in different settings, demonstrating the superiority of our HCPL in tackling scarce supervision.We also conduct similar experiments on datasets CiteSeer and PubMed following the setting (i.e., label rate) in [54].The result is shown in Tables 4 and 5 and similar results can be detected in two datasets.
Towards Effective Semi-supervised Node Classification with HCPL 82:13

Ablation Study
In this part, we perform extensive experiments over core components of the proposed HCPL.In particular, five model variants are compared with the full model, which only remove one part of our framework with the other components kept: - The results are in Table 6.First, we can see a decline in the performance of HCPL w/o aug, demonstrating the necessity of augmentation strategies, which also improve the robustness of HCPL.Second, HCPL performs better than HCPL w/o cur and HCPL inv cur, validating that pseudo-labeling may introduce some biases to deteriorate the performance while our curriculum learning strategy is capable of releasing this issue.Moreover, we analyze the homophily ratio of each dataset, which denotes the fraction of edges that connect nodes from the same category in a graph.A lower homophily ratio will increase the challenges of semi-supervised learning under label scarcity, which makes curriculum learning more important.In particular, the homophily ratios for Cora and Cite-Seer are 0.810, and 0.736, respectively.From ablation studies, it can be observed that when we remove curriculum learning, the performance will drop 1.18% and 1.61% for Cora and CiteSeer, respectively.This validates our analysis that the curriculum learning strategy is more suitable for 82:14 X. Luo et al.  challenging tasks.Third, removing the uncertainty-based selection strategy leads to a decline in performance, which shows it can produce more accurate pseudo-labels with the consideration of model robustness.Fourth, although these model variants can still perform well with the effectiveness of the remaining components, we can always observe a decline in these model variants compared with the full model, which validates the effectiveness of every component.

Sensitivity Analysis
In this part, we study the sensitivity of hyper-parameters in HCPL, i.e., embedding dimension in the hidden layer and the total number of iterations, respectively.
We first study the influence of different hidden embeddings by varying the dimension in [32,64,128,256,512,1024] with other settings fixed.We plot the result on all the datasets in Figure 4 and observe that the performance almost first increases and then stays stable as the embedding dimension grows.The potential reason is that a large hidden dimension would improve representation, but the model will tend to be saturated when the dimension is above a certain value.
Next, we study the effect of different numbers of iterations.Specifically, we fix all the other hyper-parameters and vary the iteration number in {5, 10, 20, 50}.The results are plotted in Figure 5.We can observe that in most cases increasing the iteration number leads to a gain in performance.Perhaps it is because a larger iteration number brings in fewer pseudo-labels at each iteration, which is usually reliable for self-training.However, a too-large number of iterations accompanies a higher computational cost.As a result, we set the number of iterations to 20 as the default.

Robustness Analysis
In this part, we test the robustness of our HCPL by perturbing the graph, i.e., deleting edges or masking node attributes at random. Figure 6 illustrates the performance of three methods (GCN, GAT, and HCPL) when varying the perturbation rate from 10% to 90% on three datasets Cora, Fig. 6.Robustness analysis on the datasets Cora, PubMed, and Amazon Photo in terms of classification accuracy (in %).PubMed, and Amazon Photo, respectively.It can be shown that our HCPL obtains the best results under various random attack perturbation rates.Moreover, our HCPL decreases less as the perturbation rate grows, demonstrating the robustness of our HCPL.

Efficiency Analysis
In this part, we analyze the efficiency of competing methods by comparing their running time.The compared results on six datasets are collected in Table 7. From the results, we can find that our method has better efficiency compared with various recent works (i.e., MVGCL, CG 3 , and AMGCN).Actually, besides GNNs, these current works, i.e., MVGCL, CG 3 , and AMGCN utilize additional complex techniques, which could bring in huge computational cost.MVGCL introduces different data augmentation strategies to generate multiple topological views for mutual information maximization.CG 3 needs to calculate data similarities and involves both local graph convolution and global hierarchical graph convolution.AMGCN extracts node embeddings from different views and then fuses them using the attention mechanism.These techniques bring more computational cost than our curriculum learning.Although some of the early methods have better efficiency, their performance is much worse than ours.Therefore, our HCPL exhibits competitive model scalability from the comparable running time.

Visualization
In this subsection, we demonstrate the t-SNE visualization [11] of the node embeddings generated by four methods on Cora, CiteSeer, and Amazon Photo.The compared results can be found in   From the results, the embeddings generated by our HCPL are more discriminative on these three datasets since these embeddings belonging to different categories can be better separated.This finding can result from our hybrid pseudo-labeling strategy, which provides extra high-quality supervision for the neural network, therefore validating our superiority again.

CONCLUSION
In this research, we investigate the problem of semi-supervised node classification on graphs and propose a simple yet effective model HCPL.Note that pseudo-labeling techniques are predominantly studied in the visual domains but have not been well applied to effectively solving node classification problems on graphs yet.In this article, our HCPL annotates unlabeled samples by training a classification model on the labeled nodes as well as pseudo-labeled nodes, and repeats the procedure with self-training.We propose a hybrid pseudo-label selection strategy for reliable guidance.Moreover, the concept of curriculum learning is introduced to progressively learn from simple pseudo-labels to hard pseudo-labels in terms of confidence and uncertainty.In this way, our HCPL sufficiently explore the unlabeled data through our pseudo-labeling strategy.Extensive experiments on six well-known benchmarks validate the effectiveness of the proposed HCPL.In future work, we will attempt to introduce our pseudo-labeling strategy into other graph-related tasks such as link prediction and graph classification.

Fig. 1 .
Fig.1.Illustration of our proposed approach HCPL.The model is first trained under the supervision of labeled nodes.Then, the model is used to predict and assign pseudo-labels for unlabeled nodes.We use curriculum learning to iteratively select a subset of pseudo-labeled samples based on a hybrid selection strategy, and add them to the labeled set.Afterward, a new model is re-trained under the supervision of the expanded labeled data.The process will stop when all data is exhausted during iterative training.

Fig. 2 .
Fig. 2.An example of our optimization pipeline with curriculum learning.

ALGORITHM 1 :
Training pipeline for our HCPL Input: Attribute feature set {x i } x i ∈V ; Graph edge set E; Set of labeled nodes V L ; Stepping times T ; Output: Prediction result {p i } x i ∈V U .1: Train GNN predictor using V L only; 2: for t = 1 to T do 3:

Fig. 3 .
Fig.3.Results on three datasets Cora, CiteSeer, and PubMed w.r.t. the number of iteration.We can observe that model performance improves as the number of iterations grows before saturation in most cases.
HCPL w/o aug: We delete the perturbation over the input in the GNN-based predictor.-HCPL w/o cur: We remove the curriculum strategy and annotate all the unlabeled nodes for self-training.-HCPL w/o unc: We remove the uncertainty-based selection strategy and only use confidence to select pseudo-labels.-HCPL -inv cur: We annotate from the hard examples to easy examples.-HCPL-random: We annotate samples randomly during iterations.

Figure 7 .
Figure 7. From the results, the embeddings generated by our HCPL are more discriminative on these three datasets since these embeddings belonging to different categories can be better separated.This finding can result from our hybrid pseudo-labeling strategy, which provides extra high-quality supervision for the neural network, therefore validating our superiority again.

Table 1 .
Statistics of Six Datasets

Table 2 .
Results on Six Datasets in Terms of Accuracy (in %) Over Five Runs

Table 3 .
Results on Cora Dataset with Different Label Rates in Terms of Classification Accuracies (in %)

Table 5 .
Results on PubMed Dataset with Different Label Rates in Terms of Classification Accuracies (in %)

Table 7 .
The Compared Running Time Cost of the Compared Methods (Seconds) OOM means out-of-memory.