When Measures are Unreliable: Imperceptible Adversarial Perturbations toward Top-k Multi-Label Learning

With the great success of deep neural networks, adversarial learning has received widespread attention in various studies, ranging from multi-class learning to multi-label learning. However, existing adversarial attacks toward multi-label learning only pursue the traditional visual imperceptibility but ignore the new perceptible problem coming from measures such as Precision@k and mAP@k. Specifically, when a well-trained multi-label classifier performs far below the expectation on some samples, the victim can easily realize that this performance degeneration stems from attack, rather than the model itself. Therefore, an ideal multi-labeling adversarial attack should manage to not only deceive visual perception but also evade monitoring of measures. To this end, this paper first proposes the concept of measure imperceptibility. Then, a novel loss function is devised to generate such adversarial perturbations that could achieve both visual and measure imperceptibility. Furthermore, an efficient algorithm, which enjoys a convex objective, is established to optimize this objective. Finally, extensive experiments on large-scale benchmark datasets, such as PASCAL VOC 2012, MS COCO, and NUS WIDE, demonstrate the superiority of our proposed method in attacking the top-k multi-label systems.


INTRODUCTION
With the rapid development of Deep Neural Networks (DNNs), machine learning has achieved great success in various tasks, such as image recognition [47,64], text detection [40,48], and object tracking [33,49].However, recent studies have proposed that DNNs are highly vulnerable to the well-crafted perturbations [42,50,58].Due to their intriguing vulnerability, these unguarded models usually suffer from malicious attacks and their performance can be easily affected by slight perturbations hidden behind images.Recently, many pieces of research on adversarial perturbations have found the underlying weakness in practical multimedia applications of DNNs like key text detection [21,57], video analysis [51,54], and image super-restoration [8,61].The emergence of such problems, which seriously prevent the exploration of various studies and safe applications of DNNs, has raised the attention of many researchers to search for new undetected perturbation technologies, so as to exclude more defense blind spots.
So far, researchers have developed numerous multi-class adversarial perturbation algorithms to protect systems.Most of these effective approaches have been widely adopted in real-world classification scenarios [1,22,43].In recent years, more attention has shifted to exploring potential threats in multi-label learning [34,42].In a multi-label system, common inference strategies can be divided into two categories: threshold-based ones and ranking-based ones.In the threshold-based approach, the classifier compares the label score with a pre-defined threshold to decide whether each label is relevant to the input sample.Although this strategy is intuitive, the optimal threshold is generally hard to determine.By contrast, the ranking-based approach returns the  labels with the highest scores as the final prediction, whereas the hyper-parameter  is much easier to select according to the scenario.Hence, top- multi-label learning (T  ML) has been widely applied to various multimedia-related tasks, such as information retrieval [2,4] and recommendation systems [7,27].Nevertheless, the majority of current studies on multi-label adversarial perturbations still follow the attack principle on multiclass learning, overly valuing the property of visual imperceptibility, but overlooking the new issues that come from the new scenarios.To be specific, for most single-label (multi-class/binary) classifications, when the only relevant category is misclassified, the victim may intuitively attribute the issue to the poor performance of the model, instead of considering whether the attack happens.In contrast, in a multi-label classification scenario, the victim could more likely perceive attacks when all relevant labels are ranked behind the top- position, as the classification performance of the model is well below the expectation.In view of this, we propose the concept of measure imperceptibility.Empirically, we find that it is quite easy to notice the existence of attack by common multilabel measures ranging from Accuracy, Top- Accuracy (T  Acc), to ranking-based measures such as Precision at  (P@) [31], mean Average Precision at  (mAP@) [26], and Normalized Discounted Cumulative Gain at  (NDCG@) [53].Hence, a natural question arises: Can we craft an perturbation that satisfies both (P1) visual imperceptibility and (P2) measure imperceptibility?
To answer this question, we introduce a novel algorithm to generate such perturbations for T  ML.Specifically, the core challenge lies in constructing an objective function that can disrupt the top- ranking results in a well-designed manner.To illustrate this, we provide a simple example in Figure 1, where the input image  has three relevant labels.Our goal is to generate adversarial perturbations that can prevent the classifier from detecting a specified label such as Cat.In order to meet (P1), we need to add a slight perturbation  into the image, which makes it almost no different from the original image.To satisfy (P2), we can no longer attack all relevant categories.Instead, the other relevant labels like Table and Chair should be ranked higher to compensate for the performance degeneration induced by the attack on the specified label Cat.Then, the top- measures, such as P@, mAP@, and NDCG@, will not drop significantly while the specified label has been ranked far behind the top- position.In the extreme case, the classifier can never detect certain important categories and produce degraded classification performance, which may result in serious consequences for multimedia applications, such as multimodal emotion recognition [35] and text-to-image generation [20].
Inspired by this example, we construct a novel objective that not only guarantees slight perturbations but also pursues well-designed ranking results.When optimizing this objective, both visual imperceptibility and measure imperceptibility can be achieved.Furthermore, we also propose an effective optimization algorithm to efficiently optimize the perturbation.Finally, we validate the effectiveness of our proposed method on three large-scale benchmark datasets (PASCAL VOC 2012, MS COCO 2014, and NUS WIDE).To summarize, the main contributions of this work are three-fold: • To the best of our knowledge, we are the pioneer to introduce the vital concept of measure imperceptibility in adversarial multi-label learning.• We propose a novel imperceptible attack method toward the T  ML problem, which enjoys a convex objective.And an efficient algorithm is established to optimize the proposed objective.
• Extensive experiments are conducted on large-scale benchmark datasets including PASCAL VOC 2012, MS COCO 2014, and NUS WIDE.And the empirical results show that the proposed attack method can achieve both visual and measure imperceptibility.

RELATED WORK
In this section, we will present a brief introduction to multi-class adversarial perturbations, multi-label adversarial perturbations, and top- ranking optimization.

Multi-class Adversarial Perturbations
Image-specific Perturbations is a vital part of adversarial learning, which aims to attack each specific image with a single perturbation.Szegedy et al. [45] first observes the vulnerability of DNNs and gives the corresponding robustness analysis.Then, Goodfellow et al. [16] proposes FGSM and describes the important concept of untargeted attack and targeted attack for image-specific perturbations.Inspired by these works, more perturbation methods are gradually emerging to explore these two directions.Specifically, the untargeted attack is an instance-wise attack that increases the scores of irrelevant categories to replace relevant categories in the top- set.For example, Deepfool [37], as a geometric multi-class untargeted attack, carefully crafts the small perturbation to forge the top-2 class into the correct class.On top of this, Fool [46] extends the perturbation to the top- multi-class learning, designed as the corresponding attacks to replace the ground-truth labels in the top- set with other irrelevant labels.Different from the above attacks, the targeted attack uses specific negative labels to perturb the predictions.FGSM [16] and I-FGSM [24] conduct back-propagation with the gradient of DNNs to achieve the exploration of minimum efficient perturbation.C&W [5] presents a typical multi-class adversarial learning framework under the top-1 protocol to implement the targeted attack.On the basis of C&W, CW  [63] presents an ordered top- attack by adversarial distillation framework toward top- multi-class targeted attack.Based on FGSM-based methods, MI-FGSM [12] further introduces the momentum term into an iterative algorithm to avoid coverage into the local optimum.
Our attack method has a similar effect as the untargeted attack, i.e., it replaces the relevant category in the top- prediction with other several irrelevant labels.The difference is that our perturbation only focuses on specified relevant categories to promise a slight change in measures.
Image-agnostic Perturbations is another type of emerging attack based on the untargeted attack, which dedicates to generating an overall perturbation to perturb all points.To the best of our knowledge, Moosavi-Dezfooli et al. [36] first introduces Universal Adversarial Perturbations (UAPs) method, which is independent to the single example.UAP [46], as the top- version of UAPs, focuses on top- universal adversarial learning in multi-class learning.SV-UAP [23] utilizes the singular vectors of the Jacobian matrices of the feature maps to accumulate the minimum perturbation.As interest in UAPs grows recently, a series of related approaches were carried out to further analyze the existing problem in various DNNs-based scenarios [11,25,30,60,62].

Multi-label Adversarial Perturbations
As far as we know, the first study toward multi-label adversarial attack is [42], in which the authors propose a type of targeted attack, extending image-specific perturbations to multi-label learning.In particular, ML-DP and ML-CW are the multi-label extensions of DeepFool and C&W, respectively.Besides, Melacci et al. [34] observe the importance of domain-knowledge constraints for detecting adversarial examples, which analyzes the robustness problem of untargeted attacks in multi-label learning.After realizing the importance of top-k learning in multi-label classification, Hu et al. [18] first introduces the image-specific perturbations into top- multi-label learning, where the untargeted/targeted attack attempts to replace all ground-truth labels in the top- range with arbitrary/specific  negative labels.Also, this work is the first to study the vulnerability of T  ML.
Despite the great work toward multi-class/label adversarial learning, there is no approach focused on the invisibility of perturbations for measures.By careful consideration, the previous works have been more concerned with the efficiency of the perturbation while ensuring that the perturbation is not visually recognizable.However, the occurrence of these attacks can be easily identified when we pay slight attention to the numerical changes in the measure.In this work, we thus propose an imperceptible adversarial perturbation that could easily deceive the common multi-label metrics to fill this study gap.

Top-𝑘 Ranking Optimization
Top- ranking optimization has a significant amount of applications in various fields, such as binary classification [14,15], multi-class classification [3,6,52], and multi-label learning [19].Due to the discontinuity of individual top- error, it is computationally hard to directly minimize the top- loss.Therefore, the practical solution to the top--related problem usually relies on minimizing a differentiable and convex surrogate loss function [59].In particular, one of the relevant surrogates is the average top- loss (AT  ) [14], which generalizes maximum loss and average loss to better fit the optimization in different data distributions, especially for imbalanced problems.On top of this work, Hu et al. further propose a summation version of the top- loss called "Sum of Ranked Range (SoRR)", and demonstrate its effectiveness in top- multi-label learning [19].Both of these surrogates are convex with respect to each individual loss, which means optimizing the objective function only needs simple gradient-based methods.
Inspired by these discoveries, we introduce the AT  into our algorithm to further improve the performance of our method.More details about the technology we used to optimize perturbations could be found in Sec.3.2.

PRELIMINARIES
In this section, we first give the used notation in the following content and introduce the average of top- optimization method.

Notations
Generally, we assume that the samples are drawn i.i.d from the space Z = X × Y, where X is the input space; Y = {0, 1}  is the label space with 1 for positive and 0 for negative;  represents the number of labels.In multi-label learning, each input  ∈ X is associated with a label vector  ∈ Y.We use   = { |  = 1} and   = {  |  = 0} to denote the relevant labels and irrelevant labels of , respectively, where   is the -th element of .

Average of Top-𝑘 Optimization
Top- optimization, which is a useful tool to construct our objective, has been widely applied to various tasks like information retrieval [29] and recommendation systems [32].For T  ML, it is intuitive to construct an objective function such that the lowest value of the predicted score for the relevant category is higher than the ( + 1)-th largest score [19]: And this idea has been successfully applied to generate top- multilabel adversarial perturbation [18], making the perturbed image satisfy that: max In other words, this objective expects the highest score of the relevant labels is not ranked before the top- position.However, due to the non-differentiable ranking operator, the individual top- loss is not easy to optimize [14,59].To address this issue, a series of surrogate loss functions including the sum of ranked range [19], average top- loss [14], are proposed.To be specific, for a set of  (), the average top- loss   ( ()) is defined as which is proven convex w.r.

MEASURE IMPERCEPTIBLE ATTACK
In this section, we define the concept of measure-imperceptible attack and further present the corresponding imperceptible attack algorithm.

Problem Formation
This measure-imperceptible attack attempts to attack a few categories while not degenerating the performance of ranking measures.
To achieve this goal, we need to • G1: rank the specific relevant categories lower than the -th position; • G2: rank the other relevant categories higher than the -th position as much as possible; • G3: generate an visual imperceptible perturbation  that is as smaller as possible.According to these sub-goals, we could utilize the existing symbols to form the following optimization problem: where S is the specified label set that includes partial indices of the relevant categories of a sample .
Specifically, the condition  1 guarantees that all specified categories cannot be ranked higher than the top- position,  2 requires the top- set to be filled with a certain amount of other relevant labels, and the regularizer limits the size of perturbations, respectively.These three constraints correspond to the above three sub-goals.However, it is challenging to optimize due to the complex constraints of this problem.Consequently, we expect to seek a more easily optimized formulation to solve this problem.

Optimization Relaxation
To optimize the problem in Eq.( 2), it is necessary to build a feasible optimization framework.To begin with, we could notice that for a well-trained model, most of the specified categories of an image are possibly ranked higher than the -th largest score, that is, max Meanwhile, there exist a few ground-truth labels that are not in the top- set, and they satisfy the condition that: Thus, the original problem (2) could be transformed to reduce the loss generated by combining the above two equations and the perturbation norm.Utilizing the Lagrangian equation, we obtain this surrogate loss function: where  is the trade-off hyper-parameter.However, the potential gradient sparsity problem [59] of Eq.( 3) makes the optimization very hard.To solve this problem, we could further leverage Eq.( 1) to form an equivalent convex relaxation.And the following lemma also provides us with a solution for the average top- optimization: Thus by adopting Lem.4.1 to Eq.( 1), a convex optimization framework could be established.We will show this specific relaxation process of our objective below.
Relaxation of  1 .For the sake of explanation, we first make the definitions of Δ  and Δ [ ] , which measure the difference between the maximum score of all specified labels and the -th/bottom- score of all labels, respectively, i.e., When  = 1, Δ [1] is the largest difference since  [ ] ( + ) is the smallest value among  scores.It could be known that as  increases, the value of  [ −+1] ( +) increases consistently and the difference Δ [ ] reduces.Thus, Δ [ ] refers to the -th largest element among Δ  .Then, according to the fact and Lem.4.1, we have min Furthermore, we could remove the inner redundant hinge function by the following lemma: Then, the second term in Eq.( 3) could be finally transformed into min Relaxation of  2 .Similarly, we could make the definitions of Δ and Δ[  ] to convert the third term of Eq.( 3): where Δ[  ] is the -th largest element among Δ .Correspondingly, its final optimization could be written as: Combining the above two objectives, we obtain our final objective.The final objective is shown as follows: In this way, our original single-element optimization problem is modified into a form of average top- optimization.This objective does not depend on any sorted result but just needs to calculate the sum result, which greatly reduces the complexity of optimizing perturbations.To solve this optimization problem, we could utilize the iterative gradient descent method [18,19] to directly update  and   with the following Algorithm 1.For this algorithm, we would adopt two different selection schemes to select specified categories: • S1: Global Selection.In this case, we globally specify several categories for all the samples.Note that only the samples relevant to at least one specific category are attacked.
• S2: Random Selection.In this case, we randomly specify  ground-truth labels for each example.For any integer , it satisfies that 0 <  ≤ .

EXPERIMENTS
In this section, we present the experimental settings and evaluate the quantitative results of our method against other comparison methods on three large-scale benchmark datasets and two attack schemes.

Experimental Settings
Datasets.We carry out our experiments on three well-known benchmark multi-label image annotation datasets containing PAS-CAL VOC 2012 (VOC) [13], MS COCO 2014 (COCO) [28] and NUS WIDE (NUS) [9].Considering space constraints, we present the information about these datasets in Appendix B.1 Evaluation Metrics.For each individual experiment, we would report Multi-label Top- accuracy (T  Acc) and three popular rankingbased measures for a more comprehensive comparison, including Precision at  (P@), mean Average Precision at  (mAP@) and Normalized Discounted Cumulative Gain at  (NDCG@).
With a bit of abuse of the notation, we use  to denote the indices of permutations of  ().For the instance  = (, ) and model  , Hence, the Multi-label Top- Accuracy [19] is defined as follows: where is the indicator function.Then, we make the following definition of Precision at  [31]: As an extended concept, Average Precision at  [41,56] averages the top- precision performance at different recall values: where AP  @ ( , ) refers to the AP performance on -th category.Normalized Discounted Cumulative Gain at  (NDCG@) [53] is a kind of listwise ranking measure that allocates decreasing weights to the labels from top to bottom positions, so as to precisely evaluate each ranking result: To further measure the overall effectiveness of our perturbation algorithm, it is necessary to evaluate the success rate and size of the perturbation.We now define the average attack effectiveness metric Δ, which calculates the average change number of top- specified relevant labels over all images: where S ′ is the labels in S that are still ranked in the top- region after attacking.And average perturbation (APer), which measures the average successful perturbation among all instances, is presented as: Competitors.We compare our method with the existing top- multi-label untargeted attack methods, encompassing T  ML-AP-U [18], Fool [46] and ML-CW-U [5], since their process of excluding the top- relevant labels is similar to ours.However, all of them are untargeted attacks and not for the specified categories.For the sake of fairness, we uniformly ask these methods to attack specified categories in the top- set.Then, we observe the norm of successful perturbation and its metric change.Due to length limitations, we have relegated their details to Appendix B.2. Since these competitors hardly achieve measure imperceptibility, it is unfair to require all of them to meet the strict constraints in Eq.( 2).Thus, we uniformly lower the judging standard of a successful perturbation.Note that when the number of excluded specified categories is no less than threshold  after a certain iteration, we term it a successful attack.We set  = |S| for S1, and ask this integer no larger than |S| for S2, respectively.In this case, we compare each baseline with our method by observing the value change of measures and average perturbation norm.
Implementation Details.Our experiments are implemented by PyTorch [39], running on NVIDIA TITAN RTX with CUDA v12.0.We select ResNet-50 [17] as the backbone on VOC, and ResNet-101 [17] on COCO and NUS, respectively.All parameters are initialized with the checkpoint pre-trained on ImageNet, except for the randomly initialized fully connected layer.To satisfy the multi-label classification tasks, we add the sigmoid function to keep the output results ranging from [0, 1].All pixel intensities of each RGB image range from {0, 1, . . ., 255}.More details about the models we used on each dataset are presented in Appendix B. 3.
At the attack period, all images are normalized into the range of [−1, 1].We adopt SGD [44] as our optimizer, where both learning rate and hyper-parameter  are searched within the range of {1e-3, 5e-4, 1e-4, 5e-5}, and momentum adopts 0.9.All of the victim models are well-trained so as to promise the effectiveness of our attack under various specified categories.On top of this, we only sample the instance that satisfies   ≥  + |S| from the validation set as the attack objects.The maximum number of test images under each set of parameters is set as 1000, and the maximum iteration ranges from {100, 300, 500}.

Results Analysis
The average performance of T  MIA (1) under the global selection scheme (2) on COCO and NUS are shown in Table 1 and Table 2, respectively, where the categories included in each type are specifically illustrated in Appendix B. 4. In most cases, T  MIA outperforms other comparison methods when  and |S| take different values.It could achieve a smaller change in each metric while perturbing more specified categories in S.Although Fool attacks more classes in some cases, it comes at a larger cost to generate the perturbation.Furthermore, Figure 2 clearly shows the visual effect of our attack method and other comparison methods in the global selection scheme on NUS.All competitors could manage to remove the specified class from the top- prediction.However, they raise the scores of several irrelevant categories, making them appear in the top- predictions, resulting in the performance degradation of measures.Our perturbation considers increasing the scores of other relevant categories in addition to perturbing these specified labels to achieve the effect of confusing the metrics.On the other  hand, our method achieves this goal with a smaller perturbation which could hardly be discerned.These results and analysis all demonstrate that T  MIA could achieve both visual and measure imperceptibility in the global selection scheme.
Table 3 and Table 4 present the comparative average performance of different methods under different parameters (1) in the random selection scheme (2) on COCO and NUS, respectively.For the measure performance, our perturbation performs better than other baselines.However, an interesting phenomenon is that Fool perturbs more labels while paying a huge effort, and T  ML-AP-U generates a relatively smaller perturbation but only obtains the sub-optimal metric change results.The proposed T  MIA makes a trade-off between visual imperceptibility and measure imperceptibility and achieves a better overall performance on Δ and APer.The visualization results of an example under the random selection scheme on NUS are shown in Figure 3. Comparing these value changes in metrics, we can see that T  MIA always promises a smaller change so as to make its perturbation not obvious.Even if there are still some irrelevant labels in the top- set, our method also tries to prevent them from being in the front position.This means the proposed perturbation is more favorable to maintaining the measure imperceptibility.Other experimental results and analyses on VOC, COCO, and NUS are presented in Appendix C.

CONCLUSION
In this paper, we propose a method named T  MIA to generate a novel adversarial perturbation toward top- multi-label classification models, which could satisfy both visual imperceptibility and measure imperceptibility to better evade the monitoring of defenders.To achieve this goal, we form a simple and convex objective with a corresponding gradient iterative algorithm to effectively optimize this perturbation.Finally, a series of empirical studies on different large-scale benchmark datasets and schemes are carried out.Extensive experimental results and analysis validate the effectiveness of the proposed method.Proof.We first define  as a coefficient vector with size , whose component is denoted as   .It is noted that  =1  [ ] () is the solution of the following programming problem: To solve this problem, we could utilize the Lagrangian equation, converting Eq.( 14) to where non-negative vectors , , and  ∈ R are Lagrangian multipliers.Let us take its derivative w.r.t  and set it to be 0, we obtain  =  −  + 1.According to this equation, we further get the dual problem of the original problem by substituting it into Eq.(14): Due to  ≥  − 1 ≥ 0, it is easy to know  ∈ [0, 1], and we thus obtain the final equivalent form: Through Eq.( 15), we observe  [ ] () is always an optimal solution, i.e.

□
A.2 Proof of Lemma 4.2 We carry out our experiments on three well-known benchmark multi-label image annotation datasets: • PASCAL VOC 2012 (VOC) [13]

B.2 Competitors Introductions
The details about our comparison methods are shown as follows: • T  ML-AP-U [18].This is the first top- untargeted attack method for producing multi-label adversarial perturbation, which is the main baseline to compare with our method.• Fool [46].Inspired by the traditional method DeepFool [37], this method is proposed to generate the top- adversarial perturbation in multi-class classification.To apply to multilabel learning, we consider its prediction score for the only relevant label as the maximum score among all relevant labels like [18].• ML-CW-U.We adopt a similar untargeted vision of C&W [5] as our comparison method, which adopts the loss function [max ∉    ( +) −min  ∈    ( +)] + .Compared to CW  , this method simply extends C&W to multi-label adversarial learning while not considering the order factor.

B.3 Model Details
Table 5 summarizes the parameters and mAP results of the welltrained models used on each dataset, where the momentum for SGD is empirically set as 0.9.Both ResNet-50 and ResNet-101 are pre-trained on ImageNet [10].Utilizing the binary cross-entropy loss, we further fine-tune the model to fit the classification task on different datasets.

B.4 Category Description
For the experiments under the global selection scheme on VOC, COCO, and NUS, we present the specified categories included in each type in Table 6, Table 7 and Table 8, respectively.We select various scales of specified label sets to better show the performance of our method under different parameters.

B.5 Number of Samples under Different Settings
As the selected samples have to satisfy   ≥  + |S|, the number of samples under various settings is different.Thus, we present the specific numbers of samples and their average specified labels under different settings in Table 9-12.In particular, Table 9, Table 10, and Table 11 show the correspondence under the global selection scheme on each dataset, respectively.And Table 12 integrates the number of samples on different datasets under the random selection scheme.Please note that we only select up to 1000 samples as our test images for each experiment.

C ADDITIONAL RESULTS AND ANALYSIS C.1 Overall Performance under Scheme 1
The overall performance of T  MIA and other methods under the global selection scheme on COCO, NUS, and VOC are shown in Table 13, Table 14, and Table 15, respectively.We can clearly see that our method completely outperforms the other comparison methods.Specifically, our method could generate smaller perturbations to achieve a slight change in metrics.Meanwhile, we obtain a larger Δ, which means that our approach successfully perturbs more specified categories.On one hand, we could observe that all methods generate larger perturbations with increasing the  value while the increase in our average perturbation norm is smaller than others.For example, on the Traffic type of NUS, when  changes from 2 to 5, the APer of ML-CW-U, Fool, and T  ML-AP-U increase 0.7295, 5.4391, and 0.6772, respectively.Our T  MIA only produces 0.0934 increasing.For some types (Vehicles, Animals, Household types of VOC), its APer even produces a decreasing trend.On the other hand, most changes in measures for all methods get smaller as the  value increases.But when there are large  and |S|, T  ML-AP-U yields the opposite trend, such as on Daily necessities and Furniture types of COCO.This comparison indicates that our method is more robust in complex conditions.Through these two aspects, we verify the outstanding performance of our method.

C.2 Overall Performance under Scheme 2
In the main paper, we only present the performance of different methods under Scheme 2 with the maximum iteration of 300 and  = 2 on NUS.In this part, Table 16 to Table 23 present the overall performance of different methods under various parameters on three datasets in the random selection scheme, where the maximum iteration ranges from {100, 300, 500} and  = 2, 3.
From the results on VOC and COCO, we could more intuitively find the superiority of our method.But for the results on NUS, which are shown in Table 21-23, the Δ and APer values of our perturbation are not the best.We think there are two possible reasons.The first one is that since the model training on NUS can only achieve 0.828 mAP which is shown in Table 5, its original classification effect is relatively poor.This means there exist more irrelevant labels in the top- set.To satisfy measure imperceptibility, the model needs to take more effort to push them outside the top- position; (2) as our perturbation is required to achieve both visual and measure imperceptibility, it has to make a trade-off between the perturbed number and perturbation size in some cases.We could see that Fool usually presents a better average attack effectiveness while generating an obvious perturbation, and T  ML-AP-U makes a smaller APer but overlook the impact on metrics.In contrast, our T  MIA could achieve both properties and avoid these problems.Therefore, these results and analyses further validate the effectiveness of our perturbation and optimization framework.

C.3 Visualization Results
We set the maximum iteration as 300 and  = 2 to present the image results under Scheme 1 and Scheme 2 from Figure 4 to Figure 7, where the first two sets of images are the performance on VOC and the others are on COCO.
In Figure 4, we see that for our method, the metrics of the perturbed image remain unchanged.While other methods produce a clear performance degradation that causes a significant change in metrics.In Figure 5, it is not hard to find that even if all methods achieve the ideal results, our perturbation could craft a smaller perturbation to lower the ranking position of specified labels.Hence, the overall performance of our method demonstrates that the proposed perturbation is more effective than other attacks when pursuing both visual and measure imperceptibility.
From the results on COCO, we could find that our perturbation manage to push the specified categories out of the top- region with slight efforts.Meanwhile, its changes in ranking-based metric values could be almost ignored, which achieves our perturbation effect.For Fool, its perturbations always make the perturbed images visually perceptible though it could fool all metrics in some cases.Although T  ML-AP-U performs well in some images, its overall effect is not better than our perturbation.This is consistent with our analysis shown in Sec.5.2.Table 13: The rest results of competitors and our method with the maximum iteration 300 under different  values and globally selected S on COCO, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Figure 1 :
Figure 1: The illustration of the proposed imperceptible adversarial perturbation.

Figure 2 :
Figure 2: The top-3 successful performance comparisons under global selection scheme on NUS, where the maximum iteration is 300.All perturbation intensities are magnified by a factor of 100 to enhance contrast and visibility.The specified labels, relevant labels, and irrelevant labels are marked with dark green, light green, and red, respectively.

Figure 3 :
Figure 3: The top-5 successful performance comparisons under random selection scheme on NUS, where the maximum iteration is 300.All perturbation intensities are magnified by a factor of 100 to enhance contrast and visibility.The specified labels, relevant labels, and irrelevant labels are marked with dark green, light green, and red, respectively.

Figure 4 :
Figure 4: The top-2 successful performance comparisons under global selection scheme on VOC, where the maximum iteration is 300.All perturbation intensities are magnified by a factor of 100 to enhance contrast and visibility.The specified labels, relevant labels, and irrelevant labels are marked with dark green, light green, and red, respectively.

Figure 5 :
Figure 5: The top-3 successful performance comparisons under random selection scheme on VOC, where the maximum iteration is 300.All perturbation intensities are magnified by a factor of 100 to enhance contrast and visibility.The specified labels, relevant labels, and irrelevant labels are marked with dark green, light green, and red, respectively.

Figure 6 :
Figure 6: The top-5 successful performance comparisons under global selection scheme on COCO, where the maximum iteration is 300.All perturbation intensities are magnified by a factor of 100 to enhance contrast and visibility.The specified labels, relevant labels, and irrelevant labels are marked with dark green, light green, and red, respectively.

Figure 7 :
Figure 7: The top-5 successful performance comparisons under random selection scheme on COCO, where the maximum iteration is 300.All perturbation intensities are magnified by a factor of 100 to enhance contrast and visibility.The specified labels, relevant labels, and irrelevant labels are marked with dark green, light green, and red, respectively.
t each component.For this expression, we could directly see that   ( ()) is the upper bound of  [ ] (), i.e.,   ( ()) ≥  [ ] ().Hence, this fact provides a convex equivalent form for the original top- optimization problem.

Table 1 :
The average performance with the maximum iteration 300 under different  values and globally selected S on COCO, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 2 :
The average performance with the maximum iteration 300 under globally selected S on NUS.

Table 3 :
The average performance with the maximum iteration 300 and  = 2 under randomly selected S on COCO.

Table 4 :
The average performance with the maximum iteration 300 and  = 2 under randomly selected S on NUS.

Table 5 :
Summary of well-trained models on different datasets.

Table 8 :
Correspondence between types and categories on NUS.

Table 9 :
Number of samples under different types and  values on VOC.

Table 10 :
Number of samples under different types and  values on COCO.

Table 11 :
Number of samples under different types and  values on NUS.

Table 6 :
Correspondence between types and categories on VOC.

Table 7 :
Correspondence between types and categories on COCO.

Table 12 :
Number of samples under random selection scheme.

Table 14 :
The rest results of competitors and our method with the maximum iteration 300 under different  values and globally selected S on NUS, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 15 :
The results of competitors and our method with the maximum iteration 300 under different  values and globally selected S on VOC, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 16 :
The results of competitors and our method with the maximum iteration 100 and  = 2 on VOC under different  values and sizes of randomly selected S, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 17 :
The results of competitors and our method with the maximum iteration 300 and  = 2 on VOC under different  values and sizes of randomly selected S, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 18 :
The results of competitors and our method with the maximum iteration 100 and  = 2 on COCO under different  values and sizes of randomly selected S, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 19 :
The rest results of competitors and our method with the maximum iteration 300 and  = 2 on COCO under different  values and sizes of randomly selected S, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 20 :
The results of competitors and our method with the maximum iteration 500 and  = 3 on COCO under different  values and sizes of randomly selected S, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 21 :
The results of competitors and our method with the maximum iteration 100 and  = 2 on NUS under different  values and sizes of randomly selected S, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.

Table 22 :
The rest results of competitors and our method with the maximum iteration 300 and  = 2 on NUS under different  values and sizes of randomly selected S, where Δ refers to the difference between the original value and the perturbed value of corresponding metrics.↓ means the smaller the value the better, and ↑ is the opposite.The best results under each set of parameters are bolded.